T.I.L – Page 39 – Today I Learned

ADHD

Finished this week this book. ADHD is Attention Deficit Hyperactivity Disorder. I didnt know it was like your brain madurates more slowly than normal. of Something that is getting more diagnosed these days. Maybe we were like that when younger or even our parents still are. It quite interesting to see how strict is the society when you are not 100% fit for purpose. The education system is built for a common type of child even the interaction with others. If you are not one of that class, you are going to have a difficult time and the future is not going to be very bright neither. So quite challenging to be and live with ADHD. But there are ways to move forwards if you want. The author follows the 5 Cs to deal with this “challenge”:

self-Control: If you lose your temper…. breath, meditate, step back.
Compassion: Think how the person in the other side feels.
Collaboration: Get the other side involvement in the decision making process.
Consistency: The process is not just day. It is a long, slow process.
Celebration: Yeah, acknowledge the good things. So the other Cs get stronger.

At the end of the day, you are not stupid, you have a different path for madurity and other skills.

At the same time I was reading this book, one day in the radio was this program about dyslexia. It was socking to know that about 50% of USA inmates were dyslexic… And again, it seems the end of the world… but in the program was an interview to one director from GCHQ saying that he was dyslexic and they were hiring for them. Why? Because they see and approach things in a different way.

So, at the end of the day, whatever you have, you can still move forwards in life. You dont need to be in the “normal” range of population/people.

BGP-StockMarket-EGB

I was reading through my backlog and noticed too close by incidents. A BGP hijack on 30th September from Telstra and Tokyo Stock Exchange outage on 2nd Oct. At the end of the day, small mistakes/errors (on purpose or not) can cause massive impact (depending on your point of view). For BGP, RPKI is the security framework to make sure the advertised routes belong to the real owners. Yeah, quick summary. But at the end of the day, not all Internet providers are using RPKI, and even if you use it, you can make mistakes. This is better than nothing. For the exchanges, thinking that a piece of hardware can cause a stop to a 6 trillion $ market is crazy. And it seems is just a 350 servers system. That tells me that you dont need the biggest system to hold the biggest value and you will always hit a problem no matter how safe/resilience is your design/implementation/etc. Likely I am making this up and I need to review the book, but one of the conclusions I took from it, via Godel, it doesn’t matter how many statements you use to declare your (software) system, you can always find a weakness (false statement).

Evolved-Indiana

This week I realised that Juniper JunOS was moving to Linux…. called Evolved. I guess they will still be supporting FreeBSD version but long term will be Linux. I am quite surprised as this was really announced early 2020, always late joining the party. So all big boys are running linux at some level: Cisco has done it sometime ago with nx-os, Brocade/Extrene did it too with SLX (based on Ubuntu) and obviously Arista with EOS (based on Fedora). So the trend of more “open” network OS will be on the raise.

And as well, I finished “Indiana Jones and the Temple of Doom” book. Indiana Jones films are among my favourites… although this was was always considered the “worse” (I erased from my mind the “fourth”) I have really enjoyed the book. It was like watching the movie at slow pace and didnt care that I knew the plot. I will get the other books likely.

NTS

From a new Cloudflare post, I learned that NTS is a standard. To be honest, I can’t remember there was work for making NTP secure. In the last years I have seen development in PTP for time sync in financial systems but nothing else. So it is nice to see this happening. We only need to encrypt BGP and we are done in the internet.. oh wait. Dreaming is free.

So I am trying to install and configure NTS in my system following these links: link1 link2

I have just installed ntpsec via debian packages system and that’s it, ntpsec is running…

# apt install ntpsec
...
# service ntpsec status
● ntpsec.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-10-04 20:35:58 BST; 6min ago
Docs: man:ntpd(8)
Main PID: 292116 (ntpd)
Tasks: 1 (limit: 9354)
Memory: 10.2M
CGroup: /system.slice/ntpsec.service
└─292116 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
Oct 04 20:36:02 athens ntpd[292116]: DNS: dns_check: processing 3.debian.pool.ntp.org, 8, 101
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 81.128.218.110
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 81.128.218.110
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 139.162.219.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 139.162.219.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 62.3.77.2
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 62.3.77.2
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 213.130.44.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 213.130.44.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: dns_take_status: 3.debian.pool.ntp.org=>good, 8
#

Checking the default config, there is nothing configured to use NTS so I made some changes based on the links above:

# vim /etc/ntpsec/ntp.conf
...


# Public NTP servers supporting Network Time Security:
server time.cloudflare.com:1234 nts

# Example 2: NTS-secured NTP (default NTS-KE port (123); using certificate pool of the operating system)
server ntp1.glypnod.com iburst minpoll 3 maxpoll 6 nts

#Via https://www.netnod.se/time-and-frequency/how-to-use-nts
server nts.ntp.se:3443 nts iburst
server nts.sth1.ntp.se:3443 nts iburst
server nts.sth2.ntp.se:3443 nts iburst

After restart, still not seeing NTS in sync 🙁

# service ntpsec restart
...
# ntpq -puw
remote refid st t when poll reach delay offset jitter
time.cloudflare.com .NTS. 16 0 - 64 0 0ns 0ns 119ns
ntp1.glypnod.com .NTS. 16 5 - 32 0 0ns 0ns 119ns
2a01:3f7:2:202::202 .NTS. 16 1 - 64 0 0ns 0ns 119ns
2a01:3f7:2:52::11 .NTS. 16 1 - 64 0 0ns 0ns 119ns
2a01:3f7:2:62::11 .NTS. 16 1 - 64 0 0ns 0ns 119ns
0.debian.pool.ntp.org .POOL. 16 p - 256 0 0ns 0ns 119ns
1.debian.pool.ntp.org .POOL. 16 p - 256 0 0ns 0ns 119ns
2.debian.pool.ntp.org .POOL. 16 p - 256 0 0ns 0ns 119ns
3.debian.pool.ntp.org .POOL. 16 p - 64 0 0ns 0ns 119ns
-229.191.57.185.no-ptr.as201971.net .GPS. 1 u 25 64 177 65.754ms 26.539ms 7.7279ms
+ns3.turbodns.co.uk 85.199.214.99 2 u 23 64 177 12.200ms 2.5267ms 1.5544ms
+time.cloudflare.com 10.21.8.19 3 u 25 64 177 5.0848ms 2.6248ms 2.6293ms
-ntp1.wirehive.net 202.70.69.81 2 u 21 64 177 9.6036ms 2.3986ms 1.9814ms
+ns4.turbodns.co.uk 195.195.221.100 2 u 21 64 177 10.896ms 2.9528ms 1.5288ms
-lond-web-1.speedwelshpool.com 194.58.204.148 2 u 23 64 177 5.6202ms 5.8218ms 3.2582ms
-time.shf.uk.as44574.net 85.199.214.98 2 u 29 64 77 9.0190ms 4.9419ms 2.5810ms
lux.22pf.org .INIT. 16 u - 64 0 0ns 0ns 119ns
ns1.thorcom.net .INIT. 16 u - 64 0 0ns 0ns 119ns
time.cloudflare.com .INIT. 16 u - 64 0 0ns 0ns 119ns
time.rdg.uk.as44574.net .INIT. 16 u - 64 0 0ns 0ns 119ns
-herm4.doylem.co.uk 185.203.69.150 2 u 19 64 177 15.024ms 9.5098ms 3.2011ms
-213.251.53.217 193.62.22.74 2 u 17 64 177 5.7211ms 1.4122ms 2.1895ms
*babbage.betadome.net 85.199.214.99 2 u 20 64 177 4.8614ms 4.1187ms 2.5533ms
#
#
# ntpq -c nts
NTS client sends: 56
NTS client recvs good: 0
NTS client recvs w error: 0
NTS server recvs good: 0
NTS server recvs w error: 0
NTS server sends: 0
NTS make cookies: 0
NTS decode cookies: 0
NTS decode cookies old: 0
NTS decode cookies too old: 0
NTS decode cookies error: 0
NTS KE probes good: 8
NTS KE probes_bad: 0
NTS KE serves good: 0
NTS KE serves_bad: 0
#

I ran tcpdump filtering on TCP ports 1234 (cloudflare) and 3443 (netnod), and I can see my system trying to negotiate NTS with Cloudflare and NetNod but both sessions are TCP RST 🙁

Need to carry on researching…

BPF – Linux

Last time I tried BPF was via an Ubuntu VM prepared for BPF. But this week checking another article, I realised that I can run BPF natively in my laptop!!!

So aptitude did the job installing the package, and didn’t have to install a new kernel or patch, so super easy and I can see it is working as based in the article:

# apt depends bpftrace
bpftrace
Depends: libbpfcc (>= 0.12.0)
Depends: libc6 (>= 2.27)
Depends: libclang1-9 (>= 1:9~svn359771-1~)
Depends: libgcc-s1 (>= 3.0)
Depends: libllvm9 (>= 1:9~svn298832-1~)
Depends: libstdc++6 (>= 5.2)
#
#
# dpkg -l | grep bpftrace
ii bpftrace 0.11.0-1 amd64 high-level tracing language for Linux eBPF
#
# uname -a
Linux athens 5.8.0-1-amd64 #1 SMP Debian 5.8.7-1 (2020-09-05) x86_64 GNU/Linux
#
#
# bpftrace -e 'software:faults:1 { @[comm] = count(); }'
Attaching 1 probe…
^C
@[BatteryStatusNo]: 1
@[slack]: 52
@[Xorg]: 139
@[VizCompositorTh]: 455
@[Chrome_IOThread]: 463
@[ThreadPoolForeg]: 1305
@[CompositorTileW]: 2272
@[Compositor]: 3789
@[Chrome_ChildIOT]: 4610
@[chrome]: 8020
#

And run the same script.

# bpftrace bpftrace-example.bt
Attaching 2 probes…
Sampling CPU at 99hz… Hit Ctrl-C to end.
^C
@cpu:
[0, 1) 33 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2) 23 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[2, 3) 31 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[3, 4) 23 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
#

Now I really need to play with it in my own system, no excuse...

Screen-Brightness

Another thing I realized lately was that my laptop screen was very dark, not bright at all like my external screen so it was hard to use both. I use Debian Testing LXDE as it is quite light and I dont need anything as heavy as Gnome/KDE. So I struggle how to adjust the brightness but finally got it.

I had to try different programs but finally a blog showed all possibilities and found the one that works for me.

$ brightnessctl set 800 -d intel_backlight

The next thing, I had to be sure that was effective after reboots…. So not sure if this is very clean solution, but I just added that command to my .bashrc. It works. Moving on.

VirtualBox-Python2-Debian-Dependencies

This week I realised that Debian was removing python2 support and surprisingly…. it was trying to remove VirtualBox from my system…

So it seems that VirtualBox is still depending on python2. A bit disappointing.

I am not really keen of VirtualBox but I have had to use it lately for my Kubernetes training and testing OpenBSD. I prefer using kvm/quemu. So I know I will have to workout how to do kubernetes/bsd outside VirtualBox….

Something I learned by the way was to check the dependencies of a package in Debian…. I guess it is about time.

apt-cache depends package-name

Drive

No, it is not about cars. I just finished reading Drive from Daniel Pink. I quite liked it as it is mainly focus in the daily working life. And you can find a summary at the end of the book of each chapter. Plus specific advises for different circumstances.

The books is about what is motivation, what motivate us, etc. Funny enough, again, there is a reference to “Thinking fast and slow” as a proof that we are not as rational as we think making decisions. As well there are a lot of references to “flow” from Mihaly Csikszentmihalyi. Quite interesting and central to the book too.

Initially our motivations are survival and reproduction like any other animal. That heavily changed with the Industrial revolution and the move to a workforce based in offices were the motivation was based on carrot/stick policies. That works for repetitive tasks but not for creative ones.

And I feel identified about that. I am looking for that motivation, drive in myself. I want to enjoy my job, want to learn, want to see things happening due to my actions. And I dont want a massive salary, neither bonuses as it would be more a burden that a help. Just a decent salary (I am not going to become rich working) so you can remove the money from the table and focus in what is really fulfilling. But most of the work environments are not like this. Although the books shows some punctual places where they have applied a different approach and have produced results. This one is quite radical and motivating

As well, another thing I discovered in the book, it is the term B companies. Several links about it: definition1 definition2 example1 example2

So they are for profit-companies but with some soul. Really like it. And to be honest, as a consumer, want to support that. Even maybe one day work in one of those or even set up one (related to IT, but have no idea)

The author says the new motivation/drive for this century is based on your personality. If you are not influenced much for external things, then your drive is based on: autonomy, mastery and purpose.

If your goal is external things: money, promotions, power, sex, etc. Maybe you will not have enough.

You want to take responsibility if you want to give your best so you need to have the voice to choose how, when , with whom to achieve that. You want to master your task, that’s never a quick path, but slow and sometimes hard, but that makes it worth it. And finally, you want to see a meaning for all that.

You have those 3 ingredients in your life (and they are not going to come to your), you are in a fulfilling trip.

Kubernetes Troubleshooting I

Restore ETCD

This is a process no well documented in the official docs and I messed up in my CKA exam:

1- check config of etcd process. Maybe you will need some details for the restore process

$ kubectl describe pod -n kube-system etcd-master
...
--name=master
--initial-cluster=master=https://127.0.0.1:2380
--initial-advertise-peer-urls=https://127.0.0.1:2380
...

2- Stop api-server if not running kubeadm

$ service kube-apiserver stop

3- Check help for all restore options. Keep in mind you will need (very likely) to provide certs for auth.

$ ETCDTL_API=3 etcdctl snapshot restore -h

4- Restore ETCD using a previous backup:

$ ETCDTL_API=3 etcdctl --endpoints 127.0.0.1:2379 snapshot restore FILE \
--cacert xxx --cert xx --key xxx

--data-dir /NEW/DIR \
--initial-cluster-toker TOKEN \ (token is any word) 

--name master \ 
--initial-cluster=master=https://127.0.0.1:2380 \ 
--initial-advertise-peer-urls=https://127.0.0.1:2380 

USE HTTPS!!!!

5- Add new lines and update volume paths in ETCD config. If it is a static pod, check in /etc/kubernetes/manifests in master node.

--data-dir=/NEW/DIR
--initial-cluster-token TOKEN

++ volumeMounts/volumes to new path /NEW/DIR !!!!

6- Restart services if not running kubeadm

$ systemctl daemon-reload
$ service etcd restart
$ service etcd kube-apiserver start

7- Checks

/// if using kubeadm, docker instance for etcd should restart
$ docker ps -a | grep -i etcd

/// check etcd is running showing members:
$ ETCDCTL_API=3 etcdctl member list --cacert xxx --cert xx --key xxx

Sidecar -logging

Based on this doc. You want to send some logs to stderr so you create a new container that takes those.

Container with a sidecar:

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
   args: 
   - /bin/sh 
   - -c 
   - > i=0; 
       while true; 
       do 
        echo "$i: $(date)" >> /var/log/1.log; 
        echo "$(date) INFO $i" >> /var/log/2.log; i=$((i+1)); sleep 1; 
       done 
   volumeMounts: 
   - name: varlog 
     mountPath: /var/log
  - name: sidecar-1 
    image: busybox 
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log'] 
    volumeMounts: 
      name: varlog
      mountPath: /var/log
  volumes:
    name: varlog
    emptyDir: {}

Now you can see the logs of “/var/log/1.log” going via “sidecar-1”

$ kubectl logs counter sidecar-1

CPU/Memory of a POD

Based on these links: link1 , link2, link3

If you want to use “kubectl top” you need to install “metrics-server”

$ kubectl top pod --all-namespaces

Keep in mind that “kubectl top” shows metrics for a given pod. That information is based on reports from cAdvisor, which collects real pods resource usage.

And as per link3, “kubectl top” is not the same as running “top” inside the container.

Node NotReady

Based on this link:

$ kubectl get nodes
$ kubectl describe nodes XXX

$ ssh node 
   -> check for kubelet logs 
     cat /var/log/kubelet.log
     $ journalctl -u kubelet // systemctl status kubelet --> if a service

Cocoa Peanut Butter

I like a lot nuts and peanut butter can be a good snack before/after a workout but if you try to buy a good one from the supermarket is not cheap. 500g of roasted/salted nuts is around £3 so it is easier to do it yourself and you know what is in it! I took inspiration from this blog post.

500g of peanuts (if possible unsalted)
pinch of sea salt
2 tsp of coconut oil
50g 100% cocoa
2 tsp of custer sugar

In my case, I just can find salted peanuts. So I pass them via water to remove the excess of salt.

1- Roast the peanuts in a pre-heat (200C) oven for 5 minutes. Toss them and give another a couple of minutes. Be sure they don’t burn! Let them cool for a bit until you can handle them

2- Put the peanuts, salt, coconut oil, cocoa and sugar in the food processor. Run at full speed for several minutes. Depending on your taste, you can make it super smooth. In my case, I like a bit crunchy. In the main time, taste it just in case you want to add anything else (salt, sugar, coconut oil, etc)