Category: *unix

I had a couple of blogs from cloudflare regarding DNS infrastructure. One is regarding DNS secondary and the other about having multiple DNS primary servers in a different provider.

Regarding the Secondary DNS is interesting how they have moved to a kubernetes infrastructure with its issues.

For setting up, different primary servers in a different provider makes sense nowadays with so many options. But there is some research to be done. You dont want the new primary DNS servers relaying in the same cloud provider as your current setup. It is interesting the mention using OctoDNS for zone management and TeamCity for CI/CD.

I have reading this book (1st edit – quite old) in the last months after breakfast. So I am taking some notes. I think they are things useful and I should use them. So try to write, understand, remember and use.

I am using bash 5.1.0(1)-rc3

$ bash --version
GNU bash, version 5.1.0(1)-rc3 (x86_64-pc-linux-gnu)

1- Quoting

Shell quoting: Enclose a string in single quotes unless it contains elements that you want the shell to interpolate

$ echo John has $100 note?!
John has 00 note?!
$ echo "John has $100 note?!"
John has 00 note?!
$ echo 'John has $100 note?!'
John has $100 note?!

You can’t embed a single quote inside single quotes even with a backslash. Nothing is interpolated inside single quotes. Workaround is using double quotes with escapes or, escaping a single quote outside of surrounding single quotes.

$ echo 'John doesn't have $100 notes'
^C
$
$ echo "John doesn't have $100 notes"
John doesn't have 00 notes
$
$ echo "John doesn't have \$100 notes"
John doesn't have $100 notes
$
$ echo 'John doesn'\''t have $100 notes'
John doesn't have $100 notes
$

2- Standard Output/Input

Redirect output from “ls”: It can be confusing when redirecting output from “ls” to a file and then read it as you dont see the expected format. Using “-C” you ensure the redirection will be based on “Colummns”. By default (-1), the output will be based on lines.

$ ls -ltr
total 92
-rw-r--r-- 1 tomas tomas 72533 Jul 27 2016 jabber.py
-rw-r--r-- 1 tomas tomas 16087 Jul 27 2016 anotify.py
drwxr-xr-x 2 tomas tomas 4096 Dec 10 2017 autoload
$ ls
anotify.py autoload jabber.py
$ ls > /tmp/test.txt
$ cat /tmp/test.txt    --> You would expect just one line like "ls"????
anotify.py
autoload
jabber.py
$
$ ls -C > /tmp/test2.txt
$ cat /tmp/test2.txt   --> Yes, this is the same output of standard "ls"
anotify.py autoload jabber.py
$ ls -C
anotify.py autoload jabber.py
$ ls -1
anotify.py
autoload
jabber.py
$

Redirect output and error to different files:

$ ls -ltr /boot/* > /tmp/test-out.txt 2> /tmp/test-err.txt
$
$ cat /tmp/test-out.txt
...
-rw-r--r-- 1 root root 73210069 Nov 28 11:10 /boot/initrd.img-5.9.0-1-amd64
-rw-r--r-- 1 root root 234724 Nov 28 11:10 /boot/config-5.9.0-1-amd64
/boot/grub:
total 2379
...
-rw-r--r-- 1 root root 2394102 Nov 14 18:12 unicode.pf2
-r--r--r-- 1 root root 8362 Nov 28 10:53 grub.cfg
$
$ cat /tmp/test-err.txt
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
$

Redirect output and error to same file, use “>&“. Other option is “$ program > outfile 2>&1“

$ ls -ltr /boot/* >& /tmp/test-both.txt
$
$ cat /tmp/test-both.txt
...
-rw-r--r-- 1 root root 73210069 Nov 28 11:10 /boot/initrd.img-5.9.0-1-amd64
-rw-r--r-- 1 root root 234724 Nov 28 11:10 /boot/config-5.9.0-1-amd64
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
/boot/grub:
total 2379
...
-rw-r--r-- 1 root root 2394102 Nov 14 18:12 unicode.pf2
-r--r--r-- 1 root root 8362 Nov 28 10:53 grub.cfg
$
$ ls -ltr /boot/* > /tmp/test3.txt
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
$

Grouping output from several commands: Use () for grouping them

:/tmp/aaa/bbb$ (ls -ltr; pwd; cd ..; ls -ltr; pwd) > /tmp/all.txt
:/tmp/aaa/bbb$
:/tmp/aaa/bbb$ cat /tmp/all.txt
total 0
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:28 b.txt
/tmp/aaa/bbb
total 4
drwxr-xr-x 2 tomas tomas 4096 Nov 28 16:28 bbb
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:28 aaa.txt
/tmp/aaa
:/tmp/aaa/bbb$

“tee“: read from standard input and write to standard output and files:

:/tmp/bbb$ ls -ltr
total 32
-rw-r--r-- 1 tomas tomas 30 Nov 28 15:29 test.txt
-rw-r--r-- 1 tomas tomas 32 Nov 28 15:30 test2.txt
-rw-r--r-- 1 tomas tomas 58 Nov 28 15:38 error.txt
-rw-r--r-- 1 tomas tomas 935 Nov 28 15:57 test-out.txt
-rw-r--r-- 1 tomas tomas 121 Nov 28 15:57 test-err.txt
-rw-r--r-- 1 tomas tomas 1056 Nov 28 15:59 test-both.txt
-rw-r--r-- 1 tomas tomas 935 Nov 28 16:00 test3.txt
-rw-r--r-- 1 tomas tomas 1 Nov 28 16:47 all.txt
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.a
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.aa
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.aaa
:/tmp/bbb$
:/tmp/bbb$
:/tmp/bbb$ find . -name '*.txt'
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$
:/tmp/bbb$ find . -name '.txt' | tee /tmp/tee.txt
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$
:/tmp/bbb$ cat /tmp/tee.txt
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$

Here-Doc: use \EOF to turn off shell scripting features inside here-doc.

:/tmp/aaa$ bash here.txt bb
bb $2
:/tmp/aaa$ bash here.txt aa
aa $1
:/tmp/aaa$
:/tmp/aaa$ cat here.txt
here-doc example
grep $1 <<\EOF
name note
aa $1
bb $2
cc $3
EOF
:/tmp/aaa$

“$?”return non-zero if the last command fails.

:/tmp$ pwd
/tmp
:/tmp$ echo $?
0

“&&” run next program if the preceding program worked (logic and)

:/tmp$ ./test && date
bash: ./test: No such file or directory
:/tmp$
:/tmp$ pwd && date
/tmp
Sat 28 Nov 17:20:50 GMT 2020
:/tmp$

“||” (logic or)

:/tmp/aaa$ ./test || ( printf "%b" "Failed.\n")
bash: ./test: No such file or directory
Failed.
:/tmp/aaa$

nohup: run job in background and exit shell before finishing job.

$ nohup ./long-script &

for loop:

:/tmp/aaa$ for FILE in bbb/*
do
if [ -f $FILE ]
then
cat $FILE
fi
done
a file
b file
c file
:/tmp/aaa$ ls -ltr bbb/
total 12
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:30 a.txt
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:30 b.txt
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:31 c.txt

This week I have to update a couple of yaml files and add a line in too many places. This is the chance to work smarter. So I searched how to add a line after a match and I found this link. So let’s use sed:

$ cat test.yaml 
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
  - description: R2
    route_map_in: RM-IN
$ 
$ sed '/route_map_in: RM-IN/a route_map_out: RM-OUT' test.yaml
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
route_map_out: RM-OUT
  - description: R2
    route_map_in: RM-IN
route_map_out: RM-OUT

But the I needed to add some spaces to be aligned properly… So I searched again and found this. Now try again adding the spaces we need using “\ ” for each one.

go:1.14.6|py:3.7.3|tomas@athens:~$ sed '/route_map_in: RM-IN/a \ \ \ \ route_map_out: RM-OUT' test.yaml
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
    route_map_out: RM-OUT
  - description: R2
    route_map_in: RM-IN
    route_map_out: RM-OUT

So now redirect to a new file to fully check and then you can rename:

$ sed '/route_map_in: RM-IN/a \ \ \ \ route_map_out: RM-OUT' test.yaml > test.yaml.new
$ cat test.yaml.new 
$ mv test.yaml.new test.yaml

Now it is perfect aligned. It is kernel hack, but I am happy enough and saved a lot of time copy/paste and errors for sure.

Last time I tried BPF was via an Ubuntu VM prepared for BPF. But this week checking another article, I realised that I can run BPF natively in my laptop!!!

So aptitude did the job installing the package, and didn’t have to install a new kernel or patch, so super easy and I can see it is working as based in the article:

# apt depends bpftrace
bpftrace
Depends: libbpfcc (>= 0.12.0)
Depends: libc6 (>= 2.27)
Depends: libclang1-9 (>= 1:9~svn359771-1~)
Depends: libgcc-s1 (>= 3.0)
Depends: libllvm9 (>= 1:9~svn298832-1~)
Depends: libstdc++6 (>= 5.2)
#
#
# dpkg -l | grep bpftrace
ii bpftrace 0.11.0-1 amd64 high-level tracing language for Linux eBPF
#
# uname -a
Linux athens 5.8.0-1-amd64 #1 SMP Debian 5.8.7-1 (2020-09-05) x86_64 GNU/Linux
#
#
# bpftrace -e 'software:faults:1 { @[comm] = count(); }'
Attaching 1 probe…
^C
@[BatteryStatusNo]: 1
@[slack]: 52
@[Xorg]: 139
@[VizCompositorTh]: 455
@[Chrome_IOThread]: 463
@[ThreadPoolForeg]: 1305
@[CompositorTileW]: 2272
@[Compositor]: 3789
@[Chrome_ChildIOT]: 4610
@[chrome]: 8020
#

And run the same script.

# bpftrace bpftrace-example.bt
Attaching 2 probes…
Sampling CPU at 99hz… Hit Ctrl-C to end.
^C
@cpu:
[0, 1) 33 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2) 23 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[2, 3) 31 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[3, 4) 23 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
#

Now I really need to play with it in my own system, no excuse...

Another thing I realized lately was that my laptop screen was very dark, not bright at all like my external screen so it was hard to use both. I use Debian Testing LXDE as it is quite light and I dont need anything as heavy as Gnome/KDE. So I struggle how to adjust the brightness but finally got it.

I had to try different programs but finally a blog showed all possibilities and found the one that works for me.

$ brightnessctl set 800 -d intel_backlight

The next thing, I had to be sure that was effective after reboots…. So not sure if this is very clean solution, but I just added that command to my .bashrc. It works. Moving on.

This week I realised that Debian was removing python2 support and surprisingly…. it was trying to remove VirtualBox from my system…

So it seems that VirtualBox is still depending on python2. A bit disappointing.

I am not really keen of VirtualBox but I have had to use it lately for my Kubernetes training and testing OpenBSD. I prefer using kvm/quemu. So I know I will have to workout how to do kubernetes/bsd outside VirtualBox….

Something I learned by the way was to check the dependencies of a package in Debian…. I guess it is about time.

apt-cache depends package-name

Yesterday read for first time an article about zswap. I though it was something new, but a bit search showed that it started around 2013 as per this article.

I have a 2015 Dell XPS13 with i7, 128GB SSD and 8GB RAM. But some times my systems struggle with memory and when swapping kicks in, the system gets stuck. I have used swappiness in the past but not much improvement (or it was in my former company laptop??). Anyway this is my current swappiness:

# cat /proc/sys/vm/swappiness
60
# sysctl vm.swappiness
vm.swappiness = 60

Even having “suspender” enabled in Chrome, I just have over 2GB RAM free.

$ top
top - 10:23:14 up 6 days, 1:54, 1 user, load average: 0.23, 0.41, 0.44
Tasks: 333 total, 1 running, 332 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.2 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 56.6/7933.8 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
MiB Swap: 1.3/6964.0 [| ]

Ok, check you have ZSWAP available in your kernel.

$ uname -a
Linux x 5.7.0-2-amd64 #1 SMP Debian 5.7.10-1 (2020-07-26) x86_64 GNU/Linux

$ cat /boot/config-uname -r | grep -i zswap
CONFIG_ZSWAP=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"
CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set
# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
# CONFIG_ZSWAP_DEFAULT_ON is not set

$ cat /sys/module/zswap/parameters/enabled
N

If you have “N”in the last command and “CONFIG_ZSWAP=y” then your systems supports zswap but it is not enabled.

$ echo Y | sudo tee /sys/module/zswap/parameters/enabled

$ cat /sys/module/zswap/parameters/enabled
Y

Now you can tune some parameters to increase compression rate (3:1)

# list all parameters
grep . /sys/module/zswap/parameters/*

# change  compression params
echo z3fold | sudo tee /sys/module/zswap/parameters/zpool
echo lzo | sudo tee /sys/module/zswap/parameters/compressor

How to check zswap is working? I followed this email thread to find some clues:

# cd /sys/kernel/debug/zswap

/sys/kernel/debug/zswap# grep . *
duplicate_entry:0
pool_limit_hit:0
pool_total_size:237568
reject_alloc_fail:0
reject_compress_poor:0
reject_kmemcache_fail:0
reject_reclaim_fail:0
same_filled_pages:33
stored_pages:151
written_back_pages:0

###############
## 12h later ##
###############

/sys/kernel/debug/zswap# grep . *
duplicate_entry:0
pool_limit_hit:0
pool_total_size:17944576
reject_alloc_fail:0
reject_compress_poor:8
reject_kmemcache_fail:0
reject_reclaim_fail:0
same_filled_pages:4146
stored_pages:16927
written_back_pages:0

So it seems some values are increasing.

The difficult part seems to make this change to survive a reboot. The easy way, it is just to update “/etc/rc.local”.

# cat /etc/rc.local
....
# enabling zswap - 2020-08
echo Y >/sys/module/zswap/parameters/enabled
echo z3fold >/sys/module/zswap/parameters/zpool