Category: *unix

As part of my reinstallation, I had to create a ZFS partition that I used to use for personal storage. Debian Installation process doesnt provide this option, so I have to do it manually. To be honest, it is good to remember/refresh these “basic” things, you never know when you are going to need them (urgently very likely).

As the installation process gave most of the space to the “home” partition, that’s the one I need to take space for creating my ZFS partition. I chose LVM during installation so I dont really have to deal with physical partition, it is mainly logical volumes aka “lv”.

So I rebooted in single-mode as I wanted to be sure that I didnt damage anything and I had to umount the “home” lv. So as root:

Check mounted partitions
# df -hT

Checks LV summary
# lvs

Umount /home
# umount /home/

Check "home" is not munted
# df -hT

Check VolgumeGroup summary
# vgs

Perform filesystem check before making any change
# e2fsck -fy /dev/mapper/athens--vg-home

Resize filesystem to 22G
# resize2fs /dev/mapper/athens--vg-home 22G

Check LV hasnt changed
# lvs

Reduce LV for home to 22G
# lvreduce -L 22G /dev/mapper/athens--vg-home

Check LV home is reduced
# lvs
  LV      VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home    athens-vg -wi-ao----  22.00g                                                    
  root    athens-vg -wi-ao---- <27.94g                                                    
  swap_1  athens-vg -wi-ao---- 976.00m                                                    
# 

Check you have free space in the VG
# vgs
  VG        #PV #LV #SN Attr   VSize   VFree   
  athens-vg   1   3   0 wz--n- 237.48g <186.59g
# 

Reboot to be sure everything is fine
# reboot

Check all partitions are mounted and "home" is just 22G
$ df -hT
Filesystem                  Type      Size  Used Avail Use% Mounted on
udev                        devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                       tmpfs     786M  1.6M  785M   1% /run
/dev/mapper/athens--vg-root ext4       28G  6.7G   20G  26% /
tmpfs                       tmpfs     3.9G   87M  3.8G   3% /dev/shm
tmpfs                       tmpfs     5.0M  8.0K  5.0M   1% /run/lock
/dev/sda2                   ext2      456M   72M  360M  17% /boot
/dev/mapper/athens--vg-home ext4       21G  3.0G   17G  16% /home
/dev/sda1                   vfat      496M   64M  433M  13% /boot/efi
tmpfs                       tmpfs     786M   40K  786M   1% /run/user/1000
$

Create new LV "storage" using the spare space in the VG
# lvcreate -L 186G -n storage athens-vg
  Logical volume "storage" created.
# 

Check VG space has reduced
# vgs
  VG        #PV #LV #SN Attr   VSize   VFree  
  athens-vg   1   4   0 wz--n- 237.48g 604.00m
#

Check we have a new LV storage of 186G
# lvs
  LV      VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home    athens-vg -wi-ao----  22.00g                                                    
  root    athens-vg -wi-ao---- <27.94g                                                    
  storage athens-vg -wi-a----- 186.00g                                                    
  swap_1  athens-vg -wi-ao---- 976.00m                                                    
#

Create our Zpool storage using the LV storage.
# zpool create storage /dev/mapper/athens--vg-storage 

Check Zpool status
# zpool status
  pool: storage
 state: ONLINE
config:

	NAME                  STATE     READ WRITE CKSUM
	storage               ONLINE       0     0     0
	  athens--vg-storage  ONLINE       0     0     0

errors: No known data errors
#

Check mount point for ZFS pool
# zfs get mountpoint storage
NAME     PROPERTY    VALUE       SOURCE
storage  mountpoint  /storage    default
# 

Change Zpool storage mount point to a point in my home dir
# zfs set mountpoint=/home/tomas/storage storage

Check ZFS list
# zfs list
NAME      USED  AVAIL     REFER  MOUNTPOINT
storage   165K   179G       24K  /home/yo/storage
# 

Check all partitions
$ df -hT
Filesystem                  Type      Size  Used Avail Use% Mounted on
udev                        devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                       tmpfs     786M  1.6M  785M   1% /run
/dev/mapper/athens--vg-root ext4       28G  6.7G   20G  26% /
tmpfs                       tmpfs     3.9G   87M  3.8G   3% /dev/shm
tmpfs                       tmpfs     5.0M  8.0K  5.0M   1% /run/lock
/dev/sda2                   ext2      456M   72M  360M  17% /boot
/dev/mapper/athens--vg-home ext4       21G  3.0G   17G  16% /home
/dev/sda1                   vfat      496M   64M  433M  13% /boot/efi
tmpfs                       tmpfs     786M   40K  786M   1% /run/user/1000
storage                     zfs       180G  128K  180G   1% /home/y/storage
$ 

I have used these links to refresh myself:

• lvs resize: https://www.rootusers.com/lvm-resize-how-to-decrease-an-lvm-partition/

• create lv: https://www.thegeekstuff.com/2010/08/how-to-create-lvm/
  
• create zfs pool: https://ubuntu.com/tutorials/setup-zfs-storage-pool#3-creating-a-zfs-pool

• change zfs mount point: https://docs.oracle.com/cd/E19253-01/819-5461/gaztn/index.html

To be honest, I thought I was going to struggle much more but it has been quick.

Step by step getting back to my normal environment (and trying to improve it). I said it before, I should be able to reinstall my laptop easily, like a production server….

Using mutt for sending emails via my gmail account has been something I wanted to do for a long time. After my last issue with my laptop, finally I decided to learn how to do it.

Thanks to these blogs I managed to get it working!!!

For the main setup, this link and this. For overcoming the authentication issue, this link. So you define a new password for an app in your google account as I use 2FA.

sudo aptitude install mutt

mkdir ~/.mutt

vim ~/.mutt/muttrc

This is the content of my file:

set from = "youremail@gmail.com"
set realname = "Name Surname"

# IMAP settings
set imap_user = "youremail@gmail.com"
set imap_pass = "your_new_app_password"

# SMTP settings
set smtp_url = "smtps://youremail@smtp.gmail.com"
set smtp_pass = "your_new_app_password"

# Remote Gmail folders
set folder = "imaps://imap.gmail.com/"
set spoolfile = "+INBOX"
set postponed = "+[Gmail]/Drafts"
set trash = "+[Gmail]/Trash"

# Composition
set editor = "vim"
set edit_headers = yes
set charset = UTF-8

This is the error I had before getting the app password:

$ echo "Example mutt+gmail" | mutt -s "Testing mutt+gmail" youremail@gmail.com -a test.txt
SASL authentication failed
Could not send the message.
$ 

After that. Email sent fine without error and I can see it in my inbox!

$ echo "Example mutt+gmail v2" | mutt -s "Testing mutt+gmail v2" youremail@gmail.com -a books.ods 
$ 

The only thing I dont like is I need to have a password in a text file….

So let’s use chmod so, at least, only me can read the file.

~/.mutt$ chmod og-r muttrc
~/.mutt$ ls -ltr
total 4
-rw------- 1 yy yy 687 Oct 26 23:22 muttrc
~/.mutt$ 

Although, Ideally, I would prefer to use a certificate that is only valid for gmail, but I haven’t been able to find anything related to this.

Two weeks ago I started having an issue regarding “duplicate file trigger” with some packages related to gtk-2.0 when I was doing my routine “aptitude update/ dist-upgrade”. I use Debian Testing and I have seen in the past weird things with packages but then got fixed. I understand I am in “testing” so these things may happen. Somehow, I tried to take a look If I could understand the problem and fix it. Searching didnt give much info. I found quite old entries about the issue but nothing really clarifying why I ended there. I assume that I am not the only one having these problems in the Debian community so it should be “easier” to find info from more experience people than me. But most of the cases, I rely on old data.

Somehow I “fixed” that issue but I didnt dig enough to understand what it is this thing about “triggers”. So I messed around with “/var/lib/dpkg/triggers/File” but I did a rookie mistake. I didnt do a backup of the file…. (how difficult is to type “cp File File-backup”). I remember that I had to remove several entries and they related to i386 architecture….

Well, then after a couple of days, I had a different issue. Not sure if this was triggered by my “triggers fix”.

Somehow I had some packages failing to install due to missing dependencies. It was mainly related to “python3”. So last Saturday I decided to go deep into it… And I made things worse. When you “aptitude purge” a package, you know the package and dependencies will be removed (if they are not used). While I was trying to solve my “death by dependencies” I deleted many packages that I was sure I needed. Again, I should have stopped or at least take notes of all those packages. At the end, I ended with “just” one issue. I couldnt install python3…. no idea why. In this path of destruction, I removed most of my X11 setup, so no graphic UI for me… But I didnt notice until too late. I tried to reinstall as many packages as possible and still python3 was failing. I have pyenv and I have used python3 so not sure why/how was broken. I reached a point that with so many changes, I had to reboot…. It is not the first time and after spending hours trying to fix something, that you are sure it is fixed, is still not fixed, then your reboot, and it is fixed. So I risked it, I knew that this would be fixed or would blow up big time. The later happened. And again, rookie mistake, I should have refreshed my hard drive backup, just in case you can’t access the laptop. But as my SSD looked healthy, the thought didnt cross my mind…

Reboot, got me stuck in “lightdm can’t start”….. and I couldnt login…. I was impatient. As later I learnt, I should wait a bit (long) for a time out and I would have seen my prompth.

I tried to get into rescue mode / single-user mode, and had to get by root. So I had a bit of access and I tried to run aptitude again…. I didnt have Internet access. Something I give for granted, when you login into your graphic UI, you have your nice network-manager that logs into your wifi. I always thought I was something simple like when you log into wired Internet access… I was totally wrong.

I was lucky that I had Internet in my mobile phone and that helped me a lot to move forwards with each issue. Without that, I would be still writing on paper.

So after a bit of struggling, I learned how to connect to a wifi via CLI. Some I was quite happy with that. I used this link. Thanks to the author!

1) Find out the wireless device name.
# iw dev
phy#0
	Unnamed/non-netdev interface
		wdev 0x5
		addr 60:57:18:00:9d:8a
		type P2P-device
		txpower 0.00 dBm
	Interface wlp2s0
		ifindex 2
.....

2) Check status of wireless interface.
# ip link show wlp2s0
2: wlp2s0: (BROADCAST,MULTICAST) mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff

3) Bring up interface if you dont see "UP"
# sudo ip link set wlp2s0 up  

# ip link show wlp2s0
2: wlp2s0: (NO-CARRIER,BROADCAST,MULTICAST,UP) mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff

4) Check the connection status.
# iw wlp2s0 link
Not connected.

5) Scan to find your local wifi. You need to search for the SSID that you want to connect. This can be a long sentence...
# iw wlp2s0 scan
....
BSS bb:bb:bb:bb:bb:bb(on wlp2s0) -- associated
	last seen: 136826.740s [boottime]
	TSF: 10582137299 usec (0d, 02:56:22)
	freq: 5620
	beacon interval: 100 TUs
	capability: ESS Privacy SpectrumMgmt APSD (0x0911)
	signal: -76.00 dBm
	last seen: 12 ms ago
	Information elements from Probe Response frame:
	SSID: ZZZZZZ PROVIDER SPEED PRODUCT 10Ghz
	Supported rates: 6.0* 9.0 12.0* 18.0 24.0* 36.0 48.0 54.0 
	DS Parameter set: channel 124
	HT capabilities:
		Capabilities: 0x0f
			RX LDPC
			HT20/HT40
			SM Power Save disabled
			No RX STBC
			Max AMSDU length: 3839 bytes
			No DSSS/CCK HT40
		Maximum RX AMPDU length 65535 bytes (exponent: 0x003)
		Minimum RX AMPDU time spacing: 4 usec (0x05)
		HT RX MCS rate indexes supported: 0-15, 32
		HT TX MCS rate indexes are undefined
	HT operation:
		 * primary channel: 124
		 * secondary channel offset: above
		 * STA channel width: any
		 * RIFS: 0
		 * HT protection: no
		 * non-GF present: 1
		 * OBSS non-GF present: 0
		 * dual beacon: 0
		 * dual CTS protection: 0
		 * STBC beacon: 0
		 * L-SIG TXOP Prot: 0
		 * PCO active: 0
		 * PCO phase: 0
	RSN:	 * Version: 1
		 * Group cipher: CCMP
		 * Pairwise ciphers: CCMP
		 * Authentication suites: PSK
		 * Capabilities: 1-PTKSA-RC 1-GTKSA-RC (0x0000)
......
	WPS:	 * Version: 1.0
		 * Wi-Fi Protected Setup State: 2 (Configured)
		 * Response Type: 3 (AP)
		 * UUID: bc329e00-1dd8-11b2-8601-38549b39bdd4
		 * Manufacturer: ABCD
		 * Model: ABCD GateWay
		 * Model Number: V1.0
		 * Serial Number: 12345678
		 * Primary Device Type: ABABABAB
		 * Device name: ABCD AP
		 * Config methods: PBC, Keypad
		 * RF Bands: 0x2
		 * Version2: 2.0
.....

As the blog mentions, the two important things here are SSID and RSN/WPS. I dont think it is common these days to find WEP Wifi. Connecting to a WEP it is far much easier.


6) Connect to WPA/WPA2 WiFi network: Generate wpa_supplicant config file
Generate a configuration file for wpa_supplicant that contains the pre-shared key ("passphrase") for the WiFi network.

# wpa_passphrase "ZZZZZZ PROVIDER SPEED PRODUCT 10Ghz" >> /etc/wpa_supplicant.conf 
...type in the passphrase and hit enter...

wpa_passphrase takes the SSID as the single argument. You must type in the passphrase for the WiFi network after you run the command. Using that information, wpa_passphrase will output the necessary configuration statements to the standard output. Those statements are appended to the wpa_supplicant configuration file located at /etc/wpa_supplicant.conf.

# cat /etc/wpa_supplicant.conf 
# reading passphrase from stdin
network=
	ssid="ZZZZZZ PROVIDER SPEED PRODUCT 10Ghz"
	#psk="mytest123"
	psk=aaaaaasuperlongkeybbbbbbbbbbbbbbbbbbb
}


7) Connect using wpa_supplicant config file.
# sudo wpa_supplicant -B -D wext -i wlp2s0 -c /etc/wpa_supplicant.conf

-B means run wpa_supplicant in the background.
-D specifies the wireless driver. wext is the generic driver.
-c specifies the path for the configuration file.


8) Verify you are attached to your Wifi SSID
# iw wlp2s0 link
Connected to bb:bb:bb:bb:bb:bb (on wlp2s0)
	SSID: ZZZZZZ PROVIDER SPEED PRODUCT 10Ghz
	freq: 2412
	RX: 63825 bytes (471 packets)
	TX: 1344 bytes (12 packets)
	signal: -27 dBm
	tx bitrate: 6.5 MBit/s MCS 0

	bss flags:	short-slot-time
	dtim period:	0
	beacon int:	100

9) Obtain IP address by DHCP and verify IP
# dhclient wlp2s0
#
# ip addr show wlp2s0
2: wlp2s0:  mtu 1500 qdisc mq state UP qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.3/24 brd 192.168.1.255 scope global wlp2s0
    inet6 zzz/64 scope link 
       valid_lft forever preferred_lft forever

10) Check routing / Add default if needed:
# ip route show
192.168.1.0/24 dev wlp2s0  proto kernel  scope link  src 192.168.1.3 

The above routing table contains only 1 rule which redirects all traffic destined for the local subnet (192.168.1.x) to the wlp2s0 interface. You may want to add a default routing rule to pass all other traffic through wlp2s0 as well.

# ip route add default via 192.168.1.254 dev wlp2s0
# ip route show
default via 192.168.1.254 dev wlp2s0 
192.168.1.0/24 dev wlp2s0  proto kernel  scope link  src 192.168.1.3 

11) Check external connecitivity
# ping 8.8.8.8

After being able to get Internet access I carried on troubleshooting. I was stuck with python3 issue and now ligthdm not starting up… Using this link, I could see lightdm was complaining about an issue with Xorg. So then I checked the logs of Xorg.

[    34.050] (II) LoadModule: "glx"
[    34.050] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[    34.050] (EE) Failed to load /usr/lib/xorg/modules/extensions/libglx.so: libGL.so.1: cannot open shared object file: No such file or directory
[    34.050] (EE) Failed to load module "glx" (loader failed, 0)

I searched for info about that error. I followed several pages like this and this, I installed an reinstalled the packages mentioned, but not way could fix that “glx”.

After sometime, I decided to work on the “python3” issue. I tried to follow some advice like this. Again I tried to purge as many packets as I could and reinstall everything python related. Same result, I couldn’t install python3. And weird enough, my wifi used to be disconnected whenever removed python so I had reboot and attach to the wifi a couple of times. And as well, I managed to “lock” my wifi card too, so bizarre!!! I think I followed something from here to unlock.

So I reached a point where I was totally stuck. I couldnt fix the Xorg issue and turn up the graphic desktop environment and python3 was very broken. I decided to reinstall, I was getting nowhere. I made full backup of my homedir and most of the system folders and take some logs about the current config of my system and packages currently installed so I could help me to setup the new environment quicker.

So I downloaded a netinstall iso for Debian testing. Thanks to my mobile, I managed to find the link and I “wget” it via my cli. Then I needed to install that iso in a USB so the Debian documentation was quite easy…

# cp debian-testing-amd64-netinst.iso /dev/sdb1
# sync

Ok, it was time to boot up the usb and resintall…. not that easy…. my laptop was not able to see the usb at boot time….. I checked the BIOS and USB was allowed as boot device…. Back to the shell, I could mount the USB, fdisk showed the partition had the flag enabled for booting.

I decided to find another method. I was pretty sure that in the past when creating a boot USB I had to use “dd”… so following this link:

# dd if=debian-testing-amd64-netinst.iso of=/dev/sdb1 bs=4M conv=fdatasync status=progress

The I tried to boot again, and worked!!!

So started the reinstall process. I chose to use the whole disk and the trigger a process to overwrite my whole disk…. I had to leave the laptop overnight because the process was superslow! I didnt know how to skip it….

Something I still haven’t learned is how to scroll up/down in the tty !!! During the whole troubleshooting time, it was quite frustrating because using “less” didnt work always with all my commands… I tried to find some answers like this but no joy.

Next night I follow up. Obviously the installation process didnt detect my wifi card (non-free drivers my love) and I dont have a RJ45 port… so I had to use my usb-rj45 adaptor and long rj45 cable to get back to the internet and complete the installation…

Once the process finished, you end up with a very basic system… no wifi, no Xorg, just pure CLI environment.

So started trying to install some packages I use more often and get my wifi card detected and working. As I had my backup, I could update my apt/sources.list file and get the repositories I needed.

  104  vim /etc/apt/sources.list
  105  aptitude update 
  106  aptitude dist-upgrade 
  108  aptitude install firmware-iwlwifi 
  109  lsmod | grep iw
  110  iw list
  116  reboot

I installed “lightdm” as my display manager. I finally managed to see it again! I logged in and nothing happend. I forgot to install my lightweith desktop environment… lxde.

So finally, I was back to my graphic environment. But again I forgot to install the network-manager to get my Internet connectivity sorted as I was using the CLI technique before.

One final issue was the sound card, I could see videos online but didnt have sound so I was pretty sure I was missing something again, “alsa-utils” and then with “pavucontrol” I noticed the output devices were muted.

So finally got basic functionality. I haven’t moved all my data yet as I need to create my ZFS partition so I will need a bit more tinkering in the coming days.

This is the list of installed packages so far:

aptitude install lightdm
aptitude install sudo
aptitude install gedit
aptitude install tcpdump
aptitude install nmap
aptitude install terminator
aptitude install gkrellm
aptitude install keepass2
aptitude install firefox-esr
aptitude install vagrant
aptitude install mutt
aptitude install screen
aptitude install tmux
aptitude install wpasupplicant 
aptitude install iw
aptitude install minicom
aptitude install lxde
aptitude install pavucontrol
aptitude install libreoffice
aptitude install lstopo
aptitude install apt-file
aptitude install hwloc
aptitude install parted
aptitude install ristretto 
aptitude install network-manager-gnome 
aptitude install net-tools
aptitude install vim
aptitude install git
aptitude install tig
aptitude install firmware-iwlwifi 
aptitude install qemu
aptitude install alsa-tools-gui 
aptitude install alsa-utils 
aptitude install brightnessctl 
aptitude install zfs-dkms 
aptitude install gkrellm-volume 

Things I want to learn:

• How to setup a “timeshift” (like Apple) for backup in Debian
• How to scroll in linux terminal tty
• Send emails with mutt using gmail (and with attachments) so I dont need GUI.

Things I need to setup

• Have a USB ready with a Debian Testing ISO
• How to backup my Android mobile phone and erase contents if I lose it.
• Ideally I should rebuild my laptop more frequently so I should have a PXE-like process. Maybe having some kind of manifest in my git repo. Maybe I overthinking it.

I want to count the number of interfaces that have some specific configuration in my router. I want to use the most basic tools found in linux (so dont have to assume anything else is installed) and I want to use as less commands as possible.

So this is my config:

frr version 7.5
frr defaults traditional
hostname R2
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface ens6
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
!
interface lo1
 ip router isis ISIS 
 isis passive
!
interface ens7
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
!
interface lo2
 ip router isis ISIS 
 isis passive
!
mpls ldp
 router-id 172.20.15.2
 !
 address-family ipv4
  discovery transport-address 172.20.15.2
  !
  interface ens6
  !
  interface ens7
  !
 exit-address-family
 !
!
router isis ISIS 
 net 49.0001.1720.2001.5002.00
!
line vty
!

And I want to count the number of interfaces that have “isis network point-to-point” regardless of any other config.

In this example, we have just two interfaces.

interface ens6
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point

interface ens7
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point

The pseudo-pattern should be something like:

^interface ens.*point-to-point$

So something that starts with “interface ens”, it can have anything after that and then it ends with “point-to-point”

Ideally I want to use just “grep” and it is a standard and common tool

But grep mainly works in one line each time. And my pattern covers multiple lines.

So I searched for some help and found this that uses “perl compatible regular expressions” (PCRE). I have no idea about perl but let’s give it a go:

$ grep -Pz '(?s)interface ens.*point-to-point\n' r5.txt
frr version 7.5
frr defaults traditional
hostname R2
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface ens6
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
!
interface lo1
 ip router isis ISIS 
 isis passive
!
interface ens7
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
!
interface lo2
 ip router isis ISIS 
 isis passive
!
mpls ldp
 router-id 172.20.15.2
 !
 address-family ipv4
  discovery transport-address 172.20.15.2
  !
  interface ens6
  !
  interface ens7
  !
 exit-address-family
 !
!
router isis ISIS 
 net 49.0001.1720.2001.5002.00
!
line vty
!

Let’s explain the parameters provided to grep so far:

• -P: Use perl compatible regular expressions (PCRE).
• -z: Treat the input as a set of lines, each terminated by a zero byte instead of a newline. i.e. grep treats the input as a one big line.
• (?s): activate PCRE_DOTALL, which means that ‘.’ matches any character or newline.

But if I count, we dont have the expected answer of 2:

$ grep -Pzc '(?s)interface ens.*point-to-point\n' r5.txt
1

The “z” parameter is treating the file as a single line so for grep, there is one match only. The initial command shows in bold just one block.

We notice that the pattern is matching “interface lo1” and that is not what we want, it should be ignored.

So our pattern should match the smallest string. So we want a non-greedy matching regex. So searching again, found this. It seems for Perl regex, we need to use ? after *

$ grep -Pz '(?s)interface ens.*?point-to-point\n' r5.txt
frr version 7.5
frr defaults traditiona
hostname R2
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface ens6
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
!
interface lo1
 ip router isis ISIS 
 isis passive
!
interface ens7
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
!
interface lo2
 ip router isis ISIS 
 isis passive
!
mpls ldp
 router-id 172.20.15.2
 !
 address-family ipv4
  discovery transport-address 172.20.15.2
  !
  interface ens6
  !
  interface ens7
  !
 exit-address-family
 !
!
router isis ISIS 
 net 49.0001.1720.2001.5002.00
!
line vty
!

So now, we can see two blocks highlighted. So now let’s print only the matched strings using -o:

$ grep -Pzo '(?s)interface ens.*?point-to-point\n' r5.txt
interface ens6
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
interface ens7
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point

So this looks correct but still counting (-c) doesnt work properly because -z is treating the entry as one big line.

I haven’t been able to find the solution with just one command so at the end, I have to pipe another grep. The initial grep matches the pattern, so the second one should just count a specific pattern like “point”. It should be that simple:

$ grep -Pzo '(?s)interface ens.*?point-to-point\n' r5.txt | grep point
grep: (standard input): binary file matches

Weird, I thought this was pure text but seems the ouput of the first grep has some binary data:

$ grep -Pzo '(?s)interface ens.*?point-to-point\n' r5.txt > r55.txt
$ vim r55.txt
interface ens6
 ip router isis ISIS
 isis circuit-type level-2-only
 isis network point-to-point
^@interface ens7
 ip router isis ISIS 
 isis circuit-type level-2-only
 isis network point-to-point
^@

But we can tell grep to read binary data too using -a as per this blog and then count.

$ grep -Pzo '(?s)interface ens.*?point-to-point\n' r5.txt | grep -a point
 isis network point-to-point
 isis network point-to-point
$ grep -Pzo '(?s)interface ens.*?point-to-point\n' r5.txt | grep -ac point
2

Funny enough, if I just want to count, I dont need -a:

$ grep -Pzo '(?s)interface ens.*?point-to-point\n' r5.txt | grep -c point
2

So not sure if this is the best solution but it took me a bit to find it. It seems to work:

grep -Pzo ‘(?s)interface ens.*?point-to-point\n’ r5.txt | grep -ac point

I had a couple of blogs from cloudflare regarding DNS infrastructure. One is regarding DNS secondary and the other about having multiple DNS primary servers in a different provider.

Regarding the Secondary DNS is interesting how they have moved to a kubernetes infrastructure with its issues.

For setting up, different primary servers in a different provider makes sense nowadays with so many options. But there is some research to be done. You dont want the new primary DNS servers relaying in the same cloud provider as your current setup. It is interesting the mention using OctoDNS for zone management and TeamCity for CI/CD.

I have reading this book (1st edit – quite old) in the last months after breakfast. So I am taking some notes. I think they are things useful and I should use them. So try to write, understand, remember and use.

I am using bash 5.1.0(1)-rc3

$ bash --version
GNU bash, version 5.1.0(1)-rc3 (x86_64-pc-linux-gnu)

1- Quoting

Shell quoting: Enclose a string in single quotes unless it contains elements that you want the shell to interpolate

$ echo John has $100 note?!
John has 00 note?!
$ echo "John has $100 note?!"
John has 00 note?!
$ echo 'John has $100 note?!'
John has $100 note?!

You can’t embed a single quote inside single quotes even with a backslash. Nothing is interpolated inside single quotes. Workaround is using double quotes with escapes or, escaping a single quote outside of surrounding single quotes.

$ echo 'John doesn't have $100 notes'
^C
$
$ echo "John doesn't have $100 notes"
John doesn't have 00 notes
$
$ echo "John doesn't have \$100 notes"
John doesn't have $100 notes
$
$ echo 'John doesn'\''t have $100 notes'
John doesn't have $100 notes
$

2- Standard Output/Input

Redirect output from “ls”: It can be confusing when redirecting output from “ls” to a file and then read it as you dont see the expected format. Using “-C” you ensure the redirection will be based on “Colummns”. By default (-1), the output will be based on lines.

$ ls -ltr
total 92
-rw-r--r-- 1 tomas tomas 72533 Jul 27 2016 jabber.py
-rw-r--r-- 1 tomas tomas 16087 Jul 27 2016 anotify.py
drwxr-xr-x 2 tomas tomas 4096 Dec 10 2017 autoload
$ ls
anotify.py autoload jabber.py
$ ls > /tmp/test.txt
$ cat /tmp/test.txt    --> You would expect just one line like "ls"????
anotify.py
autoload
jabber.py
$
$ ls -C > /tmp/test2.txt
$ cat /tmp/test2.txt   --> Yes, this is the same output of standard "ls"
anotify.py autoload jabber.py
$ ls -C
anotify.py autoload jabber.py
$ ls -1
anotify.py
autoload
jabber.py
$

Redirect output and error to different files:

$ ls -ltr /boot/* > /tmp/test-out.txt 2> /tmp/test-err.txt
$
$ cat /tmp/test-out.txt
...
-rw-r--r-- 1 root root 73210069 Nov 28 11:10 /boot/initrd.img-5.9.0-1-amd64
-rw-r--r-- 1 root root 234724 Nov 28 11:10 /boot/config-5.9.0-1-amd64
/boot/grub:
total 2379
...
-rw-r--r-- 1 root root 2394102 Nov 14 18:12 unicode.pf2
-r--r--r-- 1 root root 8362 Nov 28 10:53 grub.cfg
$
$ cat /tmp/test-err.txt
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
$

Redirect output and error to same file, use “>&“. Other option is “$ program > outfile 2>&1“

$ ls -ltr /boot/* >& /tmp/test-both.txt
$
$ cat /tmp/test-both.txt
...
-rw-r--r-- 1 root root 73210069 Nov 28 11:10 /boot/initrd.img-5.9.0-1-amd64
-rw-r--r-- 1 root root 234724 Nov 28 11:10 /boot/config-5.9.0-1-amd64
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
/boot/grub:
total 2379
...
-rw-r--r-- 1 root root 2394102 Nov 14 18:12 unicode.pf2
-r--r--r-- 1 root root 8362 Nov 28 10:53 grub.cfg
$
$ ls -ltr /boot/* > /tmp/test3.txt
ls: cannot open directory '/boot/efi': Permission denied
ls: cannot open directory '/boot/lost+found': Permission denied
$

Grouping output from several commands: Use () for grouping them

:/tmp/aaa/bbb$ (ls -ltr; pwd; cd ..; ls -ltr; pwd) > /tmp/all.txt
:/tmp/aaa/bbb$
:/tmp/aaa/bbb$ cat /tmp/all.txt
total 0
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:28 b.txt
/tmp/aaa/bbb
total 4
drwxr-xr-x 2 tomas tomas 4096 Nov 28 16:28 bbb
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:28 aaa.txt
/tmp/aaa
:/tmp/aaa/bbb$

“tee“: read from standard input and write to standard output and files:

:/tmp/bbb$ ls -ltr
total 32
-rw-r--r-- 1 tomas tomas 30 Nov 28 15:29 test.txt
-rw-r--r-- 1 tomas tomas 32 Nov 28 15:30 test2.txt
-rw-r--r-- 1 tomas tomas 58 Nov 28 15:38 error.txt
-rw-r--r-- 1 tomas tomas 935 Nov 28 15:57 test-out.txt
-rw-r--r-- 1 tomas tomas 121 Nov 28 15:57 test-err.txt
-rw-r--r-- 1 tomas tomas 1056 Nov 28 15:59 test-both.txt
-rw-r--r-- 1 tomas tomas 935 Nov 28 16:00 test3.txt
-rw-r--r-- 1 tomas tomas 1 Nov 28 16:47 all.txt
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.a
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.aa
-rw-r--r-- 1 tomas tomas 0 Nov 28 16:52 a.aaa
:/tmp/bbb$
:/tmp/bbb$
:/tmp/bbb$ find . -name '*.txt'
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$
:/tmp/bbb$ find . -name '.txt' | tee /tmp/tee.txt
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$
:/tmp/bbb$ cat /tmp/tee.txt
./all.txt
./test.txt
./test3.txt
./test2.txt
./test-out.txt
./error.txt
./test-both.txt
./test-err.txt
:/tmp/bbb$

Here-Doc: use \EOF to turn off shell scripting features inside here-doc.

:/tmp/aaa$ bash here.txt bb
bb $2
:/tmp/aaa$ bash here.txt aa
aa $1
:/tmp/aaa$
:/tmp/aaa$ cat here.txt
here-doc example
grep $1 <<\EOF
name note
aa $1
bb $2
cc $3
EOF
:/tmp/aaa$

“$?”return non-zero if the last command fails.

:/tmp$ pwd
/tmp
:/tmp$ echo $?
0

“&&” run next program if the preceding program worked (logic and)

:/tmp$ ./test && date
bash: ./test: No such file or directory
:/tmp$
:/tmp$ pwd && date
/tmp
Sat 28 Nov 17:20:50 GMT 2020
:/tmp$

“||” (logic or)

:/tmp/aaa$ ./test || ( printf "%b" "Failed.\n")
bash: ./test: No such file or directory
Failed.
:/tmp/aaa$

nohup: run job in background and exit shell before finishing job.

$ nohup ./long-script &

for loop:

:/tmp/aaa$ for FILE in bbb/*
do
if [ -f $FILE ]
then
cat $FILE
fi
done
a file
b file
c file
:/tmp/aaa$ ls -ltr bbb/
total 12
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:30 a.txt
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:30 b.txt
-rw-r--r-- 1 tomas tomas 7 Nov 28 17:31 c.txt

This week I have to update a couple of yaml files and add a line in too many places. This is the chance to work smarter. So I searched how to add a line after a match and I found this link. So let’s use sed:

$ cat test.yaml 
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
  - description: R2
    route_map_in: RM-IN
$ 
$ sed '/route_map_in: RM-IN/a route_map_out: RM-OUT' test.yaml
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
route_map_out: RM-OUT
  - description: R2
    route_map_in: RM-IN
route_map_out: RM-OUT

But the I needed to add some spaces to be aligned properly… So I searched again and found this. Now try again adding the spaces we need using “\ ” for each one.

go:1.14.6|py:3.7.3|tomas@athens:~$ sed '/route_map_in: RM-IN/a \ \ \ \ route_map_out: RM-OUT' test.yaml
bgp:
  bgp_as: 65000
  enable: true
  maximum_routes: 12000
  neighbors:
  - description: R1
    route_map_in: RM-IN
    route_map_out: RM-OUT
  - description: R2
    route_map_in: RM-IN
    route_map_out: RM-OUT

So now redirect to a new file to fully check and then you can rename:

$ sed '/route_map_in: RM-IN/a \ \ \ \ route_map_out: RM-OUT' test.yaml > test.yaml.new
$ cat test.yaml.new 
$ mv test.yaml.new test.yaml

Now it is perfect aligned. It is kernel hack, but I am happy enough and saved a lot of time copy/paste and errors for sure.