Bash-Cookbook-P1

I have reading this book (1st edit – quite old) in the last months after breakfast. So I am taking some notes. I think they are things useful and I should use them. So try to write, understand, remember and use.

I am using bash 5.1.0(1)-rc3

$ bash --version
GNU bash, version 5.1.0(1)-rc3 (x86_64-pc-linux-gnu)

1- Quoting

Shell quoting: Enclose a string in single quotes unless it contains elements that you want the shell to interpolate

$ echo John has $100 note?!
John has 00 note?!
$ echo "John has $100 note?!"
John has 00 note?!
$ echo 'John has $100 note?!'
John has $100 note?!

You can’t embed a single quote inside single quotes even with a backslash. Nothing is interpolated inside single quotes. Workaround is using double quotes with escapes or, escaping a single quote outside of surrounding single quotes.

$ echo 'John doesn't have $100 notes'
^C
$
$ echo "John doesn't have $100 notes"
John doesn't have 00 notes
$
$ echo "John doesn't have \$100 notes"
John doesn't have $100 notes
$
$ echo 'John doesn'\''t have $100 notes'
John doesn't have $100 notes
$

2- Standard Output/Input

Redirect output from “ls”: It can be confusing when redirecting output from “ls” to a file and then read it as you dont see the expected format. Using “-C” you ensure the redirection will be based on “Colummns”. By default (-1), the output will be based on lines.

$ ls -ltr
total 92
-rw-r--r-- 1 tomas tomas 72533 Jul 27 2016 jabber.py
-rw-r--r-- 1 tomas tomas 16087 Jul 27 2016 anotify.py
drwxr-xr-x 2 tomas tomas 4096 Dec 10 2017 autoload
$ ls
anotify.py autoload jabber.py
$ ls > /tmp/test.txt
$ cat /tmp/test.txt    --> You would expect just one line like "ls"????
anotify.py
autoload
jabber.py
$
$ ls -C > /tmp/test2.txt
$ cat /tmp/test2.txt   --> Yes, this is the same output of standard "ls"
anotify.py autoload jabber.py
$ ls -C
anotify.py autoload jabber.py
$ ls -1
anotify.py
autoload
jabber.py
$

Redirect output and error to different files:

$ ls -ltr /boot/* > /tmp/test-out.txt 2> /tmp/test-err.txt
$
$ cat /tmp/test-out.txt
...
-rw-r--r-- 1 root root 73210069 Nov 28 11:10 /boot/initrd.img-5.9.0-1-amd64
-rw-r--r-- 1 root root 234724 Nov 28 11:10 /boot/config-5.9.0-1-amd64
/boot/grub:
total 2379
...
-rw-r--r-- 1 root root 2394102 Nov 14 18:12 unicode.pf2
-r--r--r-- 1 root root 8362 Nov 28 10:53 grub.cfg
$
$ cat /tmp/test-err.txt
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
$

Redirect output and error to same file, use “>&“. Other option is “$ program > outfile 2>&1

$ ls -ltr /boot/* >& /tmp/test-both.txt
$
$ cat /tmp/test-both.txt
...
-rw-r--r-- 1 root root 73210069 Nov 28 11:10 /boot/initrd.img-5.9.0-1-amd64
-rw-r--r-- 1 root root 234724 Nov 28 11:10 /boot/config-5.9.0-1-amd64
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
/boot/grub:
total 2379
...
-rw-r--r-- 1 root root 2394102 Nov 14 18:12 unicode.pf2
-r--r--r-- 1 root root 8362 Nov 28 10:53 grub.cfg
$
$ ls -ltr /boot/* > /tmp/test3.txt
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
$

Grouping output from several commands: Use () for grouping them

:/tmp/aaa/bbb$ (ls -ltr; pwd; cd ..; ls -ltr; pwd) > /tmp/all.txt
:/tmp/aaa/bbb$
:/tmp/aaa/bbb$ cat /tmp/all.txt
total 0
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:28 b.txt
/tmp/aaa/bbb
total 4
drwxr-xr-x 2 tomas tomas 4096 Nov 28 16:28 bbb
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:28 aaa.txt
/tmp/aaa
:/tmp/aaa/bbb$

tee“: read from standard input and write to standard output and files:

:/tmp/bbb$ ls -ltr
total 32
-rw-r--r-- 1 tomas tomas 30 Nov 28 15:29 test.txt
-rw-r--r-- 1 tomas tomas 32 Nov 28 15:30 test2.txt
-rw-r--r-- 1 tomas tomas 58 Nov 28 15:38 error.txt
-rw-r--r-- 1 tomas tomas 935 Nov 28 15:57 test-out.txt
-rw-r--r-- 1 tomas tomas 121 Nov 28 15:57 test-err.txt
-rw-r--r-- 1 tomas tomas 1056 Nov 28 15:59 test-both.txt
-rw-r--r-- 1 tomas tomas 935 Nov 28 16:00 test3.txt
-rw-r--r-- 1 tomas tomas 1 Nov 28 16:47 all.txt
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.a
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.aa
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.aaa
:/tmp/bbb$
:/tmp/bbb$
:/tmp/bbb$ find . -name '*.txt'
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$
:/tmp/bbb$ find . -name '.txt' | tee /tmp/tee.txt
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$
:/tmp/bbb$ cat /tmp/tee.txt
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$

Here-Doc: use \EOF to turn off shell scripting features inside here-doc.

:/tmp/aaa$ bash here.txt bb
bb $2
:/tmp/aaa$ bash here.txt aa
aa $1
:/tmp/aaa$
:/tmp/aaa$ cat here.txt
here-doc example
grep $1 <<\EOF
name note
aa $1
bb $2
cc $3
EOF
:/tmp/aaa$

“$?”return non-zero if the last command fails.

:/tmp$ pwd
/tmp
:/tmp$ echo $?
0

“&&” run next program if the preceding program worked (logic and)

:/tmp$ ./test && date
bash: ./test: No such file or directory
:/tmp$
:/tmp$ pwd && date
/tmp
Sat 28 Nov 17:20:50 GMT 2020
:/tmp$

“||” (logic or)

:/tmp/aaa$ ./test || ( printf "%b" "Failed.\n")
bash: ./test: No such file or directory
Failed.
:/tmp/aaa$

nohup: run job in background and exit shell before finishing job.

$ nohup ./long-script &

for loop:

:/tmp/aaa$ for FILE in bbb/*
do
if [ -f $FILE ]
then
cat $FILE
fi
done
a file
b file
c file
:/tmp/aaa$ ls -ltr bbb/
total 12
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:30 a.txt
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:30 b.txt
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:31 c.txt

sed

This week I have to update a couple of yaml files and add a line in too many places. This is the chance to work smarter. So I searched how to add a line after a match and I found this link. So let’s use sed:


$ cat test.yaml 
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
  - description: R2
    route_map_in: RM-IN
$ 
$ sed '/route_map_in: RM-IN/a route_map_out: RM-OUT' test.yaml
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
route_map_out: RM-OUT
  - description: R2
    route_map_in: RM-IN
route_map_out: RM-OUT

But the I needed to add some spaces to be aligned properly… So I searched again and found this. Now try again adding the spaces we need using “\ ” for each one.

go:1.14.6|py:3.7.3|tomas@athens:~$ sed '/route_map_in: RM-IN/a \ \ \ \ route_map_out: RM-OUT' test.yaml
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
    route_map_out: RM-OUT
  - description: R2
    route_map_in: RM-IN
    route_map_out: RM-OUT

So now redirect to a new file to fully check and then you can rename:

$ sed '/route_map_in: RM-IN/a \ \ \ \ route_map_out: RM-OUT' test.yaml > test.yaml.new
$ cat test.yaml.new 
$ mv test.yaml.new test.yaml

Now it is perfect aligned. It is kernel hack, but I am happy enough and saved a lot of time copy/paste and errors for sure.

Evolved-Indiana

This week I realised that Juniper JunOS was moving to Linux…. called Evolved. I guess they will still be supporting FreeBSD version but long term will be Linux. I am quite surprised as this was really announced early 2020, always late joining the party. So all big boys are running linux at some level: Cisco has done it sometime ago with nx-os, Brocade/Extrene did it too with SLX (based on Ubuntu) and obviously Arista with EOS (based on Fedora). So the trend of more “open” network OS will be on the raise.

And as well, I finished “Indiana Jones and the Temple of Doom” book. Indiana Jones films are among my favourites… although this was was always considered the “worse” (I erased from my mind the “fourth”) I have really enjoyed the book. It was like watching the movie at slow pace and didnt care that I knew the plot. I will get the other books likely.

NTS

From a new Cloudflare post, I learned that NTS is a standard. To be honest, I can’t remember there was work for making NTP secure. In the last years I have seen development in PTP for time sync in financial systems but nothing else. So it is nice to see this happening. We only need to encrypt BGP and we are done in the internet.. oh wait. Dreaming is free.

So I am trying to install and configure NTS in my system following these links: link1 link2

I have just installed ntpsec via debian packages system and that’s it, ntpsec is running…

# apt install ntpsec
...
# service ntpsec status
● ntpsec.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-10-04 20:35:58 BST; 6min ago
Docs: man:ntpd(8)
Main PID: 292116 (ntpd)
Tasks: 1 (limit: 9354)
Memory: 10.2M
CGroup: /system.slice/ntpsec.service
└─292116 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
Oct 04 20:36:02 athens ntpd[292116]: DNS: dns_check: processing 3.debian.pool.ntp.org, 8, 101
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 81.128.218.110
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 81.128.218.110
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 139.162.219.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 139.162.219.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 62.3.77.2
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 62.3.77.2
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool taking: 213.130.44.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: Pool poking hole in restrictions for: 213.130.44.252
Oct 04 20:36:02 athens ntpd[292116]: DNS: dns_take_status: 3.debian.pool.ntp.org=>good, 8
#

Checking the default config, there is nothing configured to use NTS so I made some changes based on the links above:

# vim /etc/ntpsec/ntp.conf
...


# Public NTP servers supporting Network Time Security:
server time.cloudflare.com:1234 nts

# Example 2: NTS-secured NTP (default NTS-KE port (123); using certificate pool of the operating system)
server ntp1.glypnod.com iburst minpoll 3 maxpoll 6 nts

#Via https://www.netnod.se/time-and-frequency/how-to-use-nts
server nts.ntp.se:3443 nts iburst
server nts.sth1.ntp.se:3443 nts iburst
server nts.sth2.ntp.se:3443 nts iburst

After restart, still not seeing NTS in sync πŸ™

# service ntpsec restart
...
# ntpq -puw
remote refid st t when poll reach delay offset jitter
time.cloudflare.com .NTS. 16 0 - 64 0 0ns 0ns 119ns
ntp1.glypnod.com .NTS. 16 5 - 32 0 0ns 0ns 119ns
2a01:3f7:2:202::202 .NTS. 16 1 - 64 0 0ns 0ns 119ns
2a01:3f7:2:52::11 .NTS. 16 1 - 64 0 0ns 0ns 119ns
2a01:3f7:2:62::11 .NTS. 16 1 - 64 0 0ns 0ns 119ns
0.debian.pool.ntp.org .POOL. 16 p - 256 0 0ns 0ns 119ns
1.debian.pool.ntp.org .POOL. 16 p - 256 0 0ns 0ns 119ns
2.debian.pool.ntp.org .POOL. 16 p - 256 0 0ns 0ns 119ns
3.debian.pool.ntp.org .POOL. 16 p - 64 0 0ns 0ns 119ns
-229.191.57.185.no-ptr.as201971.net .GPS. 1 u 25 64 177 65.754ms 26.539ms 7.7279ms
+ns3.turbodns.co.uk 85.199.214.99 2 u 23 64 177 12.200ms 2.5267ms 1.5544ms
+time.cloudflare.com 10.21.8.19 3 u 25 64 177 5.0848ms 2.6248ms 2.6293ms
-ntp1.wirehive.net 202.70.69.81 2 u 21 64 177 9.6036ms 2.3986ms 1.9814ms
+ns4.turbodns.co.uk 195.195.221.100 2 u 21 64 177 10.896ms 2.9528ms 1.5288ms
-lond-web-1.speedwelshpool.com 194.58.204.148 2 u 23 64 177 5.6202ms 5.8218ms 3.2582ms
-time.shf.uk.as44574.net 85.199.214.98 2 u 29 64 77 9.0190ms 4.9419ms 2.5810ms
lux.22pf.org .INIT. 16 u - 64 0 0ns 0ns 119ns
ns1.thorcom.net .INIT. 16 u - 64 0 0ns 0ns 119ns
time.cloudflare.com .INIT. 16 u - 64 0 0ns 0ns 119ns
time.rdg.uk.as44574.net .INIT. 16 u - 64 0 0ns 0ns 119ns
-herm4.doylem.co.uk 185.203.69.150 2 u 19 64 177 15.024ms 9.5098ms 3.2011ms
-213.251.53.217 193.62.22.74 2 u 17 64 177 5.7211ms 1.4122ms 2.1895ms
*babbage.betadome.net 85.199.214.99 2 u 20 64 177 4.8614ms 4.1187ms 2.5533ms
#
#
# ntpq -c nts
NTS client sends: 56
NTS client recvs good: 0
NTS client recvs w error: 0
NTS server recvs good: 0
NTS server recvs w error: 0
NTS server sends: 0
NTS make cookies: 0
NTS decode cookies: 0
NTS decode cookies old: 0
NTS decode cookies too old: 0
NTS decode cookies error: 0
NTS KE probes good: 8
NTS KE probes_bad: 0
NTS KE serves good: 0
NTS KE serves_bad: 0
#

I ran tcpdump filtering on TCP ports 1234 (cloudflare) and 3443 (netnod), and I can see my system trying to negotiate NTS with Cloudflare and NetNod but both sessions are TCP RST πŸ™

Need to carry on researching…

BPF – Linux

Last time I tried BPF was via an Ubuntu VM prepared for BPF. But this week checking another article, I realised that I can run BPF natively in my laptop!!!

So aptitude did the job installing the package, and didn’t have to install a new kernel or patch, so super easy and I can see it is working as based in the article:

# apt depends bpftrace
bpftrace
Depends: libbpfcc (>= 0.12.0)
Depends: libc6 (>= 2.27)
Depends: libclang1-9 (>= 1:9~svn359771-1~)
Depends: libgcc-s1 (>= 3.0)
Depends: libllvm9 (>= 1:9~svn298832-1~)
Depends: libstdc++6 (>= 5.2)
#
#
# dpkg -l | grep bpftrace
ii bpftrace 0.11.0-1 amd64 high-level tracing language for Linux eBPF
#
# uname -a
Linux athens 5.8.0-1-amd64 #1 SMP Debian 5.8.7-1 (2020-09-05) x86_64 GNU/Linux
#
#
# bpftrace -e 'software:faults:1 { @[comm] = count(); }'
Attaching 1 probe…
^C
@[BatteryStatusNo]: 1
@[slack]: 52
@[Xorg]: 139
@[VizCompositorTh]: 455
@[Chrome_IOThread]: 463
@[ThreadPoolForeg]: 1305
@[CompositorTileW]: 2272
@[Compositor]: 3789
@[Chrome_ChildIOT]: 4610
@[chrome]: 8020
#

And run the same script.

# bpftrace bpftrace-example.bt
Attaching 2 probes…
Sampling CPU at 99hz… Hit Ctrl-C to end.
^C
@cpu:
[0, 1) 33 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2) 23 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[2, 3) 31 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[3, 4) 23 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
#
Now I really need to play with it in my own system, no excuse...

Screen-Brightness

Another thing I realized lately was that my laptop screen was very dark, not bright at all like my external screen so it was hard to use both. I use Debian Testing LXDE as it is quite light and I dont need anything as heavy as Gnome/KDE. So I struggle how to adjust the brightness but finally got it.

I had to try different programs but finally a blog showed all possibilities and found the one that works for me.

$ brightnessctl set 800 -d intel_backlight

The next thing, I had to be sure that was effective after reboots…. So not sure if this is very clean solution, but I just added that command to my .bashrc. It works. Moving on.

VirtualBox-Python2-Debian-Dependencies

This week I realised that Debian was removing python2 support and surprisingly…. it was trying to remove VirtualBox from my system…

So it seems that VirtualBox is still depending on python2. A bit disappointing.

I am not really keen of VirtualBox but I have had to use it lately for my Kubernetes training and testing OpenBSD. I prefer using kvm/quemu. So I know I will have to workout how to do kubernetes/bsd outside VirtualBox….

Something I learned by the way was to check the dependencies of a package in Debian…. I guess it is about time.

apt-cache depends package-name

ZSWAP

Yesterday read for first time an article about zswap. I though it was something new, but a bit search showed that it started around 2013 as per this article.

I have a 2015 Dell XPS13 with i7, 128GB SSD and 8GB RAM. But some times my systems struggle with memory and when swapping kicks in, the system gets stuck. I have used swappiness in the past but not much improvement (or it was in my former company laptop??). Anyway this is my current swappiness:

# cat /proc/sys/vm/swappiness
60
# sysctl vm.swappiness
vm.swappiness = 60

Even having “suspender” enabled in Chrome, I just have over 2GB RAM free.

$ top
top - 10:23:14 up 6 days, 1:54, 1 user, load average: 0.23, 0.41, 0.44
Tasks: 333 total, 1 running, 332 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.2 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 56.6/7933.8 [||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
MiB Swap: 1.3/6964.0 [| ]

Ok, check you have ZSWAP available in your kernel.

$ uname -a
Linux x 5.7.0-2-amd64 #1 SMP Debian 5.7.10-1 (2020-07-26) x86_64 GNU/Linux

$ cat /boot/config-uname -r | grep -i zswap
CONFIG_ZSWAP=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"
CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set
# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
# CONFIG_ZSWAP_DEFAULT_ON is not set

$ cat /sys/module/zswap/parameters/enabled
N

If you have “N”in the last command and “CONFIG_ZSWAP=y” then your systems supports zswap but it is not enabled.

$ echo Y | sudo tee /sys/module/zswap/parameters/enabled

$ cat /sys/module/zswap/parameters/enabled
Y

Now you can tune some parameters to increase compression rate (3:1)

# list all parameters
grep . /sys/module/zswap/parameters/*

# change  compression params
echo z3fold | sudo tee /sys/module/zswap/parameters/zpool
echo lzo | sudo tee /sys/module/zswap/parameters/compressor

How to check zswap is working? I followed this email thread to find some clues:

# cd /sys/kernel/debug/zswap

/sys/kernel/debug/zswap# grep . *
duplicate_entry:0
pool_limit_hit:0
pool_total_size:237568
reject_alloc_fail:0
reject_compress_poor:0
reject_kmemcache_fail:0
reject_reclaim_fail:0
same_filled_pages:33
stored_pages:151
written_back_pages:0

###############
## 12h later ##
###############

/sys/kernel/debug/zswap# grep . *
duplicate_entry:0
pool_limit_hit:0
pool_total_size:17944576
reject_alloc_fail:0
reject_compress_poor:8
reject_kmemcache_fail:0
reject_reclaim_fail:0
same_filled_pages:4146
stored_pages:16927
written_back_pages:0

So it seems some values are increasing.

The difficult part seems to make this change to survive a reboot. The easy way, it is just to update “/etc/rc.local”.

# cat /etc/rc.local
....
# enabling zswap - 2020-08
echo Y >/sys/module/zswap/parameters/enabled
echo z3fold >/sys/module/zswap/parameters/zpool

journalctl -fu

I rebooted my laptop today and realised that docker wasnt running… It was running before the reboot and I didn’t upgrade anything related to docker (or I thought)

$ docker ps -a
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
$

Let’s check status and start if needed:

root@athens:/var/log# service docker status
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-08-21 08:34:03 BST; 7min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 12015 (code=exited, status=1/FAILURE)
Aug 21 08:34:03 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 21 08:34:03 athens systemd[1]: Stopped Docker Application Container Engine.
Aug 21 08:34:03 athens systemd[1]: docker.service: Start request repeated too quickly.
Aug 21 08:34:03 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:34:03 athens systemd[1]: Failed to start Docker Application Container Engine.
Aug 21 08:34:42 athens systemd[1]: docker.service: Start request repeated too quickly.
Aug 21 08:34:42 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:34:42 athens systemd[1]: Failed to start Docker Application Container Engine.
root@athens:/var/log#
root@athens:/var/log#
root@athens:/var/log# service docker start
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
root@athens:/var/log# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-08-21 08:41:20 BST; 5s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Process: 35305 ExecStart=/usr/sbin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
Main PID: 35305 (code=exited, status=1/FAILURE)
Aug 21 08:41:19 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Aug 21 08:41:19 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:41:19 athens systemd[1]: Failed to start Docker Application Container Engine.
Aug 21 08:41:20 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 21 08:41:20 athens systemd[1]: Stopped Docker Application Container Engine.
Aug 21 08:41:20 athens systemd[1]: docker.service: Start request repeated too quickly.
Aug 21 08:41:20 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:41:20 athens systemd[1]: Failed to start Docker Application Container Engine.
root@athens:/var/log#

Ok, so not much info… let check the recommend details:

root@athens:/var/log# journalctl -xe
β–‘β–‘ Support: https://www.debian.org/support
β–‘β–‘
β–‘β–‘ A start job for unit docker.socket has begun execution.
β–‘β–‘
β–‘β–‘ The job identifier is 4236.
Aug 21 08:41:20 athens systemd[1]: Listening on Docker Socket for the API.
β–‘β–‘ Subject: A start job for unit docker.socket has finished successfully
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://www.debian.org/support
β–‘β–‘
β–‘β–‘ A start job for unit docker.socket has finished successfully.
β–‘β–‘
β–‘β–‘ The job identifier is 4236.
Aug 21 08:41:20 athens systemd[1]: docker.service: Start request repeated too quickly.
Aug 21 08:41:20 athens systemd[1]: docker.service: Failed with result 'exit-code'.
β–‘β–‘ Subject: Unit failed
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://www.debian.org/support
β–‘β–‘
β–‘β–‘ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Aug 21 08:41:20 athens systemd[1]: Failed to start Docker Application Container Engine.
β–‘β–‘ Subject: A start job for unit docker.service has failed
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://www.debian.org/support
β–‘β–‘
β–‘β–‘ A start job for unit docker.service has finished with a failure.
β–‘β–‘
β–‘β–‘ The job identifier is 4113 and the job result is failed.
Aug 21 08:41:20 athens systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
β–‘β–‘ Subject: Unit failed
β–‘β–‘ Defined-By: systemd
β–‘β–‘ Support: https://www.debian.org/support
β–‘β–‘
β–‘β–‘ The unit docker.socket has entered the 'failed' state with result 'service-start-limit-hit'.
root@athens:/var/log# systemctl status docker.service log
Unit log.service could not be found.
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-08-21 08:41:20 BST; 1min 2s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Process: 35305 ExecStart=/usr/sbin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
Main PID: 35305 (code=exited, status=1/FAILURE)
Aug 21 08:41:19 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Aug 21 08:41:19 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:41:19 athens systemd[1]: Failed to start Docker Application Container Engine.
Aug 21 08:41:20 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 21 08:41:20 athens systemd[1]: Stopped Docker Application Container Engine.
Aug 21 08:41:20 athens systemd[1]: docker.service: Start request repeated too quickly.
Aug 21 08:41:20 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:41:20 athens systemd[1]: Failed to start Docker Application Container Engine.
root@athens:/var/log#

So “journalctl -xe” and “systemctl status docker.service log” gave nothing useful….

So I searched for “docker.socket: Failed with result ‘service-start-limit-hit'” as it was the message that looked more suspicious. I landed here and tried one command to get more logs that I didnt know: “journaltctl -fu docker”

root@athens:/var/log# journalctl -fu docker
-- Logs begin at Sun 2020-02-02 21:12:23 GMT. --
Aug 21 08:42:41 athens dockerd[35469]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaStat
Aug 21 08:42:41 athens dockerd[35469]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaEntry
Aug 21 08:42:41 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Aug 21 08:42:41 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:42:41 athens systemd[1]: Failed to start Docker Application Container Engine.
Aug 21 08:42:41 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 21 08:42:41 athens systemd[1]: Stopped Docker Application Container Engine.
Aug 21 08:42:41 athens systemd[1]: docker.service: Start request repeated too quickly.
Aug 21 08:42:41 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:42:41 athens systemd[1]: Failed to start Docker Application Container Engine.
Aug 21 08:44:32 athens systemd[1]: Starting Docker Application Container Engine…
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.Metrics
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.HugetlbStat
Aug 21 08:44:32 athens dockerd[35538]: unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '"' after object key:value pair
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.PidsStat
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.CPUStat
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.CPUUsage
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.Throttle
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.MemoryStat
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.MemoryEntry
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.BlkIOStat
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.BlkIOEntry
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaStat
Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaEntry
Aug 21 08:44:32 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Aug 21 08:44:32 athens systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 21 08:44:32 athens systemd[1]: Failed to start Docker Application Container Engine.
Aug 21 08:44:32 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Aug 21 08:44:32 athens systemd[1]: Stopped Docker Application Container Engine.

And now, yes, I could see the docker logs properly… and found the culprit and fixed. I am pretty sure the last time I played with “/etc/docker/daemon.json” I restarted docker and it was fine…

Anyway, I learned a new command “journaltctl -fu SERVICE” to troubleshoot services.

Forward TCPDump to Wireshark

Reading this blog entry I realised that very likely I have never tried forward tcpdump to a wireshark. How many times I have taken a pcap in a switch and then I need to download to see the details in wireshark…

I guess you can find some blocking points in firewalls (at least for 2-steps option)

So I tried the single command with a switch in my ceoslab and it works!

Why it works?

ssh <username>@<switch>Β  "bash tcpdump -s 0 -U -n -w - -i <interface>" | wireshark -k -i -

The ssh command is actually executing the “bash tcpdump…” remotely. But the key is the “-U” and “-w -” flags. “-U” in conjunction with “-w” sends the packet without waiting for the buffer to fill. Then “-w -” says that it writes the output to stdout instead of a file. If you run the command without -U, it would work but it will update a bit slower as it needs to fill the buffers.

From tcpdump manual:

       -U
       --packet-buffered
              If the -w option is not specified, make the printed packet output ``packet-buffered''; i.e., as the description of the contents of each packet is printed, it will be written to the standard  output,  rather  than, when not writing to a terminal, being written only when the output buffer fills.

              If  the  -w  option  is  specified, make the saved raw packet output ``packet-buffered''; i.e., as each packet is saved, it will be written to the output file, rather than being written only when the output buffer fills.

              The -U flag will not be supported if tcpdump was built with an older version of libpcap that lacks the pcap_dump_flush() function.

......
   -w file
          Write the raw packets to file rather than parsing and printing them out.  They can later be printed with the -r option.  Standard output is used if file is ``-''.

          This  output will be buffered if written to a file or pipe, so a program reading from the file or pipe may not see packets for an arbitrary amount of time after they are received.  Use the -U flag to cause packets to be written as soon as they are received.

And the stdout of that process is the ssh command so we redirect that outout via a pipe “|” and it is sent as input for wireshark thanks to “-i -” that makes wireshark to read from stdin (that is the stdout from the tcpdump in the switch!)

The wireshark manual:

       -i|--interface  <capture interface>|-
           Set the name of the network interface or pipe to use for live packet capture.

           Network interface names should match one of the names listed in "wireshark -D" (described above); a number, as reported by "wireshark -D", can also be used.  If you're using UNIX, "netstat -i", "ifconfig -a" or "ip link" might also work to list interface names, although not all versions of UNIX support the -a flag to ifconfig.

           If no interface is specified, Wireshark searches the list of interfaces, choosing the first non-loopback interface if there are any non-loopback interfaces, and choosing the first loopback interface if there are no non-loopback interfaces.  If there are no interfaces at all, Wireshark reports an error and doesn't start the capture.

           Pipe names should be either the name of a FIFO (named pipe) or "-" to read data from the standard input.  On Windows systems, pipe names must be of the form "\\pipe\.\pipename".  Data read from pipes must be in standard pcapng or pcap format. Pcapng data must have the same endianness as the capturing host.

           This option can occur multiple times. When capturing from multiple interfaces, the capture file will be saved in pcapng format.

....

       -k  Start the capture session immediately.  If the -i flag was specified, the capture uses the specified interface.  Otherwise, Wireshark searches the list of interfaces, choosing the first non-loopback interface if
           there are any non-loopback interfaces, and choosing the first loopback interface if there are no non-loopback interfaces; if there are no interfaces, Wireshark reports an error and doesn't start the capture.

The two-steps option relies on “nc” to send/receive the data, but it is the same idea regarding the tcpdump/wireshark flags using “-“

On switch: tcpdump -s 0 -U -n -w - -i <interface> | nc <computer-ip> <port>
On PC: netcat -l -p <port> | wireshark -k -S -i -