The Great Suspender

No, it is not me when I was a kid. It is a GC extension. I have a very bad habit of opening many tabs in my browser with the excuse, I will take a look later. That takes a big toll in CPU/Memory. With this extension, my laptop is running very smoothly even when I have three cEOS docker boxes running in the background. The fan runs less often. I have been using it for over a week and I am very happy with it. Need to find something for Firefox.

Github + ssh-key

There are many links for this in the Internet so I am not going to discover the fire but I struggled a bit so….

The official links from github were ok and other people did a very good job too documenting the process.

I had already a key that I wanted to use. So adding it to the repo was ok.

Testing it was my challenge. I was missing two things. My key wasn’t following the standard file name so it wasn’t used by my ssh-agent and then, i wasn’t using the “git” user when testing…. I was using my github username.

So add the key and check it is there.

$ ssh-add ~/.ssh/id_ed25519-gh
$ ssh-add -l -E md5
256 MD5:xx:xx:xx:xx:xx:67:xx:6a:73:xx:8a:xx:7f:78:xx:xx user@gh (ED25519)

Check you can ssh to github.

$ ssh -T
Hi xxxx! You've successfully authenticated, but GitHub does not provide shell access.

Ok, all good now. But this is not a new repo, how I move from the “old” user/pass to the “new” ssh-key process?

You can clone the repo again using ssh:

Or you can change the git config locally in the “url” bit.

/ceos-testing/.git master$ cat config
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
#url =
url =
fetch = +refs/heads/:refs/remotes/origin/
[branch "master"]
remote = origin
merge = refs/heads/master

After that you can “git push” using your ssh-key.

Which SSH keyfile was used to authenticate a login?

I have realised that I had two keys in my VPS and I wasn’t sure which one it was used when I was ssh-ing so I had to search a bit to find out.

These two links cover the process:

1- You need to increase the logging of your sshd (destination – server)

server# vim /etc/ssh/sshd_config
server# service sshd restart
server# tail -f /var/log/auth.log

2- From client, just ssh as usual to the server and check auth.log as per above

Jul 3 14:17:55 server sshd[8600]: Connection from IPV6 port 57628 on IPV6::453 port 64022
Jul 3 14:17:55 server sshd[8600]: Postponed publickey for client from IPv6 port 57628 ssh2 [preauth]
Jul 3 14:17:55 server sshd[8600]: Accepted publickey for client from IPv6 port 57628 ssh2: ED25519 SHA256:BtOAX9eVpFJJgJ5HzjKU8E973m+MX+3gDxsm7eT/iEQ
Jul 3 14:17:55 server sshd[8600]: pam_unix(sshd:session): session opened for user client by (uid=0)
Jul 3 14:17:55 server sshd[8600]: User child is on pid 8606
Jul 3 14:17:55 server sshd[8606]: Starting session: shell on pts/7 for client from IPv6 port 57628 id 0

3- So we have the fingertip of the key used by client. Now we need to get the fingertips of our clients keys to find the match:

client $ ssh-keygen -l -f ~/.ssh/
256 SHA256:BtOAX9eVpFJJgJ5HzjKU8E973m+MX+3gDxsm7eT/iEQ client@local (ED25519)

4- So the we can see that I am using my key to connect to the server


Nornir is a python framework mainly for network automation. Instead of using another tool like Ansible (that you need to learn), you can do the same just using pure python all the way. Ansible doesnt scale well and can be very slow, with nornir you have threading from day zero, so if you have to run tasks in 100 devices, you will feel and see the difference.

I learnt about nornir via Kirk Byers’ course. Unfortunately I didnt have the chance/time to use it in my former day job so now I have had time to review things and do a small project.

From in the nornir section you can find the whole environment. I tested on the 3-node topology.

It is nothing special. The script builds the config for BGP or ISIS using jinj2 and yaml files. I have the feeling that my jinja2 is a bit difficult to follow. Then using napalm connects to the devices to push or check the config.

Just one issue, as it seems due to the nature of cEOS relaying on docker and my filesystem, if you decide to push the config (dry_run=False == commit=True) the task will fail (while trying to write startup config) but it is actually executed.

(testdir2) /testdir2/ceos-testing/nornir master$ python -b isis -c
hostname: r1
task: deploy_config for isis
failed: True
logs: Traceback (most recent call last):
File ".../testdir2/lib/python3.7/site-packages/pyeapi/", line 469, in send
raise CommandError(code, msg, command_error=err, output=out)
pyeapi.eapilib.CommandError: Error [1000]: CLI command 5 of 5 'write memory' failed: could not run command [Error copying system:/running-config to flash:/startup-config (Operation not permitted)]
changed: False

hostname: r2
task: deploy_config for isis
failed: False
logs: None
changed: False

hostname: r3
task: deploy_config for isis
failed: False
logs: None
changed: False

This shouldn’t happen on vEOS or the real hardware (if you have the correct aaa config of course)

Route Reflectors – Notes

client -> clients and non-clients
non-client -> clients
No Reflect:
non-client -> no-client (normal ibgp)

always advertises to eBGP peers (normal ebgp)
eBGP learned prefixes, advertised to client and non-clients (normal ebgp)

Full Mesh iBGP:

  • between RRs
  • RRs and non-clients
  • clients just need iBGP to RRs.

Reflects ONLY the best route

RRs dont modify on reflected routes: NEXT_HOP, AS_PATH, LP and MED.

Prevent routing information loops: ORIGINATOR_ID and CLUSTER_LIST


  • ORIGINATOR_ID: The first RR creates the Originator_ID and sets it to the BGP router ID of the router that originates the route. So when a client receives a route with its own Originator ID, it is dropped.
  • CLUSTER_LIST: if the local CLUSTER_ID is found in the list, the route is discarded.
    *This is done ONLY in RRs.
    This is ONLY created or updated on a RR during Reflection.
Hierarchical Route Reflection:
2 levels:
- level1 RRs are clients of level2 RR -> level1 RR dont need full mesh between them
- level2 RRs are full mesh between them.

size of top-level mesh
number of alternative paths

Hierarchical Route Reflection:
2 levels:

– level1 RRs are clients of level2 RR -> level1 RR dont need full mesh between them
– level2 RRs are full mesh between them.


  • size of top-level mesh
  • number of alternative paths

RR Design Priciples:

  • keep logical and physical topologies congruent to increase redundancy and path optimization and prevent loops
    — follow physical topology
    — change physical topology
    — modify logical topology
    — follow physical topology
    — session between RR and non-client shouldnt traverse a client
    — session between RR and client shouldnt traverse a non-client
  • use comparable metrics in route selection to avoid convergence oscillations
    (ie MED – only used between prefixes from same neighbor AS – not used for different AS)
    — full ibgp mess –> no
    — always-compare-med –> no
    — deterministic-med -> ok
    — med=0 (via RM in) -> ok
    — bgp communities ->
  • set proper intra and inter cluster IGP metrics to avoid convergence oscillations
    — multicluster RR architecture: intracluster metrics lower than intercluster.
  • modify next-hop with care. Do so only to bring the RRs into forwarding path.
  • use peer groups with RR to reduce convergence time.

ASCII recording

It is something I have watched from several sites and finally noticed a page that mentioned how was done.

So this is the tool:

And this is an example I have recorded. I have just to create a HTML entry and copy the code.


$ python -m pip install asciinema
$ asciinema rec  /// or ///  python -m asciiema rec
asciinema: recording asciicast to /tmp/tmpvjba98p7-ascii.cast
asciinema: press or type "exit" when you're done
$ exit
asciinema: recording finished
asciinema: press to upload to, to save locally
asciinema: asciicast saved to /tmp/tmphxkq5sp2-ascii.cast
$ asciinema upload /tmp/tmphxkq5sp2-ascii.cast

GCP – Basic Debian VM

Template for getting a basic Debian VM with python virtualenv and pyenv. Run python3 by default

$ sudo aptitude update
$ sudo aptitude dist-upgrade

$ sudo apt-get install -y python3-pip
$ sudo apt install git
$ sudo adduser --home /home/USER --shell /bin/bash USER
$ sudo usermod -a -G sudo USER

$ vim .bashrc

Python configuration
git clone ~/.pyenv
git clone ~/.pyenv/plugins/pyenv-virtualenvwrapper
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if [ ! -d "$PYENV_ROOT" ];then
git clone ~/.pyenv
if [ ! -d "$PYENV_ROOT/plugins/pyenv-virtualenvwrapper" ]; then
mkdir -p $PYENV_ROOT/plugins
git clone $PYENV_ROOT/plugins/pyenv-virtualenvwrapper
if type "pyenv" &> /dev/null; then
eval "$(pyenv init -)"
# TODO: make the prompt work for python and ruby
__pyversion (){
if type "python" > /dev/null; then
pyenv_python_version=$(pyenv version | sed -e 's/ .*//')
printf $pyenv_python_version
if pyenv which pip &> /dev/null; then pyenv virtualenvwrapper fi export PS1="py:\$(__pyversion)|$PS1"
export PROJECT_HOME=~/git
end python

$ bash

$ sudo apt-get install -y --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev tcpdump tree

$ pyenv install 3.7.3
$ pyenv global 3.7.3

GCP Networking 101 – IP Forwarding

I had my shiny and tiny GCP network for EVE-NG to test vEOS. I built a new VM (vm2) to be my center for automation so I can test stuff like ansible/napalm/nornir etc… But I couldn’t ping from vm2 to the vEOS instances in eve-ng (vm1). Those instances where in a different network attached to vm1 so it had to “route”.

As usual, I missed one step when I created the EVE-NG VM. The official documentation doesnt mention anything regarding enabling routing in the VM. As I am not used to Cloud environments, I assume that any simple Linux VM can forward traffic if configured.

Surprise Surprise. In GCP (not sure in other cloud providers), you need to enable “forwarding” during the VM creation and you can’t change that afterwards in any way.

After checking the second guide I followed, I realised that guide mentioned the point to enable forwarding to avoid the same problem I was facing…

So I had to gave up and had to build both VMs from scratch….

But at the end, I have routing enabled in both VMs and I can ping to the vEOS images.

And another annoying thing. I couldnt update the next hop in a static route defined in the VPC. So I had to delete it and create again pointing to the new VM with the vEOS.

And dealing with the internal IPs…

Moving on, quite frustrating day. But learned several things about GCP netwoking.


I was reading a chat today, and people were talking about issues with EH. As usual, I didn’t pay attention to the very beginning of the conversation. At the end, after reading the initial link from the conversation, this was all related to IPv6 Extended Headers. And it seems they can cause issues even showed in a rfc7872.

This is the agenda from “NPS/CAIDA 2020 Virtual IPv6 Workshop” last week. And this Geoff Huston’s presentation that started the conversation. And one more link from Geoff about measuring IPv6.

And this is an old issue about IPv6, fragmentation, load balancers, anycast networks that was very interesting to read. IPv6 MTU is 1280.

I don’t have production experience with IPv6 so I try to learn from others. At some point I need to create a proper IPv6 lab with IPv6 services (NTP, DNS, DHPC, HTTPs, etc)

BPF: basics

I had in my to-try list BPF. After reading this link, I realised that it was quite straightforward to get a vagrant box with all the requirements.

As per link recommends, watch the video. I didnt know Brendan Gregg worked for Netflix and that Facebook was pushing very hard in BPF development.

You can find all the documentation here:

So as per the main link:

git clone && cd bpftracing
vagrant up

Once you have installed the vagrant box (it takes a while) you can “vagrant halt” and start again:

~/storage/technology/linux/bpftracing master$ vagrant status
Current machine states:
default poweroff (virtualbox)
The VM is powered off. To restart the VM, simply run vagrant up
~/storage/technology/linux/bpftracing master$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider…
==> default: Checking if box 'ubuntu/bionic64' version '20200525.0.0' is up to date…
==> default: Clearing any previously set forwarded ports…
==> default: Clearing any previously set network interfaces…
==> default: Preparing network interfaces based on configuration…
default: Adapter 1: nat
==> default: Forwarding ports…
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations…
==> default: Booting VM…
==> default: Waiting for machine to boot. This may take a few minutes…
default: SSH address:
default: SSH username: vagrant
default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM…
default: The guest additions on this VM do not match the installed version of
default: VirtualBox! In most cases this is fine, but in rare cases it can
default: prevent things such as shared folders from working properly. If you see
default: shared folder errors, please make sure the guest additions within the
default: virtual machine match the version of VirtualBox you have installed on
default: your host and reload your VM.
default: Guest Additions Version: 5.2.34
default: VirtualBox Version: 6.1
==> default: Mounting shared folders…
default: /vagrant => /home/xxx/storage/technology/linux/bpftracing
==> default: Machine already provisioned. Run vagrant provision or use the --provision
==> default: flag to force provisioning. Provisioners marked to run always will still run.
~/storage/technology/linux/bpftracing master$ vagrant ssh
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-106-generic x86_64)
System information as of Sun Jun 21 19:25:26 UTC 2020
System load: 0.35 Processes: 99
Usage of /: 32.2% of 9.63GB Users logged in: 0
Memory usage: 12% IP address for enp0s3:
Swap usage: 0%
0 packages can be updated.
0 updates are security updates.
Last login: Sun Jun 21 19:22:37 2020 from
vagrant@ubuntu-bionic:~$ cd /vagrant/
vagrant@ubuntu-bionic:/vagrant$ ls
Makefile Vagrantfile bpf_program.o monitor-exec bpf_program.c loader.c ubuntu-bionic-18.04-cloudimg-console.log

You can find tools (under /usr/sbin)(already compiled and ready to use) or examples (under /usr/share/doc/bpfcc-tools/examples)

These are the tools you can find in the system:

vagrant@ubuntu-bionic:~$ ls -ltr /usr/sbin | grep -i bpfcc
-rwxr-xr-x 1 root root 3496 Nov 29 2017 reset-trace-bpfcc
-rwxr-xr-x 1 root root 7105 Nov 29 2017 deadlock_detector.c-bpfcc
-rwxr-xr-x 1 root root 9029 Mar 27 2018 zfsslower-bpfcc
-rwxr-xr-x 1 root root 5131 Mar 27 2018 zfsdist-bpfcc
-rwxr-xr-x 1 root root 8184 Mar 27 2018 xfsslower-bpfcc
-rwxr-xr-x 1 root root 4431 Mar 27 2018 xfsdist-bpfcc
-rwxr-xr-x 1 root root 6825 Mar 27 2018 wakeuptime-bpfcc
-rwxr-xr-x 1 root root 2636 Mar 27 2018 vfsstat-bpfcc
-rwxr-xr-x 1 root root 1177 Mar 27 2018 vfscount-bpfcc
-rwxr-xr-x 1 root root 2978 Mar 27 2018 ttysnoop-bpfcc
-rwxr-xr-x 1 root root 31977 Mar 27 2018 trace-bpfcc
-rwxr-xr-x 1 root root 4159 Mar 27 2018 tplist-bpfcc
-rwxr-xr-x 1 root root 17766 Mar 27 2018 tcptracer-bpfcc
-rwxr-xr-x 1 root root 9327 Mar 27 2018 tcptop-bpfcc
-rwxr-xr-x 1 root root 5631 Mar 27 2018 tcpretrans-bpfcc
-rwxr-xr-x 1 root root 11996 Mar 27 2018 tcplife-bpfcc
-rwxr-xr-x 1 root root 6858 Mar 27 2018 tcpconnlat-bpfcc
-rwxr-xr-x 1 root root 6963 Mar 27 2018 tcpconnect-bpfcc
-rwxr-xr-x 1 root root 5782 Mar 27 2018 tcpaccept-bpfcc
-rwxr-xr-x 1 root root 12809 Mar 27 2018 syscount-bpfcc
-rwxr-xr-x 1 root root 1231 Mar 27 2018 syncsnoop-bpfcc
-rwxr-xr-x 1 root root 4560 Mar 27 2018 statsnoop-bpfcc
-rwxr-xr-x 1 root root 15860 Mar 27 2018 stackcount-bpfcc
-rwxr-xr-x 1 root root 6244 Mar 27 2018 sslsniff-bpfcc
-rwxr-xr-x 1 root root 6277 Mar 27 2018 solisten-bpfcc
-rwxr-xr-x 1 root root 4048 Mar 27 2018 softirqs-bpfcc
-rwxr-xr-x 1 root root 3409 Mar 27 2018 slabratetop-bpfcc
-rwxr-xr-x 1 root root 5643 Mar 27 2018 runqlen-bpfcc
-rwxr-xr-x 1 root root 5998 Mar 27 2018 runqlat-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 rubystat-bpfcc
-rwxr-xr-x 1 root root 60 Mar 27 2018 rubyobjnew-bpfcc
-rwxr-xr-x 1 root root 56 Mar 27 2018 rubygc-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 rubyflow-bpfcc
-rwxr-xr-x 1 root root 59 Mar 27 2018 rubycalls-bpfcc
-rwxr-xr-x 1 root root 60 Mar 27 2018 pythonstat-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 pythongc-bpfcc
-rwxr-xr-x 1 root root 60 Mar 27 2018 pythonflow-bpfcc
-rwxr-xr-x 1 root root 61 Mar 27 2018 pythoncalls-bpfcc
-rwxr-xr-x 1 root root 9831 Mar 27 2018 profile-bpfcc
-rwxr-xr-x 1 root root 1139 Mar 27 2018 pidpersec-bpfcc
-rwxr-xr-x 1 root root 57 Mar 27 2018 phpstat-bpfcc
-rwxr-xr-x 1 root root 57 Mar 27 2018 phpflow-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 phpcalls-bpfcc
-rwxr-xr-x 1 root root 4858 Mar 27 2018 opensnoop-bpfcc
-rwxr-xr-x 1 root root 2337 Mar 27 2018 oomkill-bpfcc
-rwxr-xr-x 1 root root 11141 Mar 27 2018 offwaketime-bpfcc
-rwxr-xr-x 1 root root 10464 Mar 27 2018 offcputime-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 nodestat-bpfcc
-rwxr-xr-x 1 root root 56 Mar 27 2018 nodegc-bpfcc
-rwxr-xr-x 1 root root 9289 Mar 27 2018 nfsslower-bpfcc
-rwxr-xr-x 1 root root 4587 Mar 27 2018 nfsdist-bpfcc
-rwxr-xr-x 1 root root 3221 Mar 27 2018 mysqld_qslower-bpfcc
-rwxr-xr-x 1 root root 12023 Mar 27 2018 mountsnoop-bpfcc
-rwxr-xr-x 1 root root 17963 Mar 27 2018 memleak-bpfcc
-rwxr-xr-x 1 root root 2262 Mar 27 2018 mdflush-bpfcc
-rwxr-xr-x 1 root root 3429 Mar 27 2018 llcstat-bpfcc
-rwxr-xr-x 1 root root 3295 Mar 27 2018 killsnoop-bpfcc
-rwxr-xr-x 1 root root 61 Mar 27 2018 javathreads-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 javastat-bpfcc
-rwxr-xr-x 1 root root 60 Mar 27 2018 javaobjnew-bpfcc
-rwxr-xr-x 1 root root 56 Mar 27 2018 javagc-bpfcc
-rwxr-xr-x 1 root root 58 Mar 27 2018 javaflow-bpfcc
-rwxr-xr-x 1 root root 59 Mar 27 2018 javacalls-bpfcc
-rwxr-xr-x 1 root root 5154 Mar 27 2018 hardirqs-bpfcc
-rwxr-xr-x 1 root root 3852 Mar 27 2018 gethostlatency-bpfcc
-rwxr-xr-x 1 root root 7124 Mar 27 2018 funcslower-bpfcc
-rwxr-xr-x 1 root root 7442 Mar 27 2018 funclatency-bpfcc
-rwxr-xr-x 1 root root 12448 Mar 27 2018 funccount-bpfcc
-rwxr-xr-x 1 root root 5847 Mar 27 2018 filetop-bpfcc
-rwxr-xr-x 1 root root 7235 Mar 27 2018 fileslower-bpfcc
-rwxr-xr-x 1 root root 3718 Mar 27 2018 filelife-bpfcc
-rwxr-xr-x 1 root root 9605 Mar 27 2018 ext4slower-bpfcc
-rwxr-xr-x 1 root root 5674 Mar 27 2018 ext4dist-bpfcc
-rwxr-xr-x 1 root root 5944 Mar 27 2018 execsnoop-bpfcc
-rwxr-xr-x 1 root root 20036 Mar 27 2018 deadlock_detector-bpfcc
-rwxr-xr-x 1 root root 3920 Mar 27 2018 dcstat-bpfcc
-rwxr-xr-x 1 root root 4009 Mar 27 2018 dcsnoop-bpfcc
-rwxr-xr-x 1 root root 3780 Mar 27 2018 dbstat-bpfcc
-rwxr-xr-x 1 root root 7130 Mar 27 2018 dbslower-bpfcc
-rwxr-xr-x 1 root root 12614 Mar 27 2018 cpuunclaimed-bpfcc
-rwxr-xr-x 1 root root 4975 Mar 27 2018 cpudist-bpfcc
-rwxr-xr-x 1 root root 57 Mar 27 2018 cobjnew-bpfcc
-rwxr-xr-x 1 root root 4142 Mar 27 2018 capable-bpfcc
-rwxr-xr-x 1 root root 6960 Mar 27 2018 cachetop-bpfcc
-rwxr-xr-x 1 root root 4932 Mar 27 2018 cachestat-bpfcc
-rwxr-xr-x 1 root root 9887 Mar 27 2018 btrfsslower-bpfcc
-rwxr-xr-x 1 root root 6214 Mar 27 2018 btrfsdist-bpfcc
-rwxr-xr-x 1 root root 2392 Mar 27 2018 bpflist-bpfcc
-rwxr-xr-x 1 root root 1721 Mar 27 2018 bitesize-bpfcc
-rwxr-xr-x 1 root root 6171 Mar 27 2018 biotop-bpfcc
-rwxr-xr-x 1 root root 4869 Mar 27 2018 biosnoop-bpfcc
-rwxr-xr-x 1 root root 4023 Mar 27 2018 biolatency-bpfcc
-rwxr-xr-x 1 root root 1567 Mar 27 2018 bashreadline-bpfcc
-rwxr-xr-x 1 root root 33534 Mar 27 2018 argdist-bpfcc

You can run them like this:

root@ubuntu-bionic:/# cpudist-bpfcc
Tracing on-CPU time… Hit Ctrl-C to end.
usecs : count distribution
0 -> 1 : 4 | |
2 -> 3 : 0 | |
4 -> 7 : 33 |* |
8 -> 15 : 24 | |
16 -> 31 : 49 |* |
32 -> 63 : 74 |** |
64 -> 127 : 161 |* | 128 -> 255 : 101 | |
256 -> 511 : 76 | |
512 -> 1023 : 116 |* | 1024 -> 2047 : 64 |** | 2048 -> 4095 : 178 |* | 4096 -> 8191 : 558 | |
8192 -> 16383 : 191 | |
16384 -> 32767 : 288 | | 32768 -> 65535 : 338 | |
65536 -> 131071 : 756 |* |
131072 -> 262143 : 1025 || 262144 -> 524287 : 950 |*** |

This link is a bit old (you need to add -bpfcc to all commands) but I think it can give you some more details for some tools.

In general, this is quite beyond my knowledge. But I would like to be able to set it up in Debian and obviously use it.