This week Job Snijders advertised the latest version of openbsd. I have been always a dreamer of being a hacker (like the movies) and the best guys when I was in Uni were Linux users. I had no idea what Linux/Unix/BSD was at that time. At the end (by the 4th year in Uni) I managed to install Linux in my windows PC without destroying anything. And fortunately, I have been using it since then. Learning more and still fortunately, in the last 6 years, using it everyday at work too.
Still very very far away from being a hacker though 🙂
In this time, I have read a bit about the BSD vs Linux threads about licenses, security, etc. And actually I was always keen to learn a bit. In Motorola, I had to use Solaris (even managed to get a certification!).
So this week, I tried to setup a VM in my debian laptop for using OpenBSD 6.7
I found and followed this link, so all credits for the author.
First I downloaded openbsd 6.7 (install67.iso) from here. There are many mirrors around the world. Prepare the file:
/var/lib/libvirt/images# ls -ltr
total 1386064
-rwxr--r-- 1 libvirt-qemu libvirt-qemu 996671488 Apr 6 2018 debian-VAGRANTSLASH-stretch64_vagrant_box_image_9.1.0.img
-rwxr--r-- 1 libvirt-qemu libvirt-qemu 950796288 Nov 29 23:17 centos-VAGRANTSLASH-7_vagrant_box_image_1905.1.img
-rw-r--r-- 1 ss ss 470118400 May 21 23:23 install67.iso
root@athens:/var/lib/libvirt/images# chown libvirt-qemu.libvirt-qemu install67.iso
/var/lib/libvirt/images# mv install67.iso openbsd67.iso
Something that confused my was that I was installing openbsd6.7 but the os-variant in the command must be obenbsd5.8. Anything else, fails.
In my setup, I have virt-viewer installed so it opened up and finished the installation using that.
I was surprised how quick was everything and didnt find any problem:
Once I logged in, I felt useless 🙂 I used a bit the shell and tested I could ssh from my host pc to the openbsd vm.
So now, I can find a book of openbsd for dummies and get going!
So close virt-viewer and stop the VM:
/var/lib/libvirt/images# virsh
virsh # list
Id Name State
2 openbsd running
virsh #
virsh #
virsh # destroy openbsd
Domain openbsd destroyed
virsh # list
Id Name State
virsh #
virsh # list --all
Id Name State
openbsd shut off
virsh #
Test we can start up again:
# virsh
Welcome to virsh, the virtualisation interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # list --all
Id Name State
openbsd shut off
virsh # start openbsd
Domain openbsd started
virsh # list --all
Id Name State
3 openbsd running
virsh # exit
#
# virt-viewer
I have just finished “Deep Work” from Cal Newport. For a long time I have believed that multitasking is the best thing to be productive but with the years passing by I realised that like a computer, context switching is very expensive on me. You can’t really concentrate in some demanding thing and then try to be on top of small things and interrupted by everybody. I am happy that I am not use social media but still at work I am easily distracted by people demands, emails, etc.
And I am pretty sure that It is not just me. Most people suffer this. And to be honest, I want to improve, I want to make a more meaningful job with my time. And life (like Winifred Gallagher) Cal’s examples (himself and others) are really good. I liked quite a lot the one regarding Daniel Kilov and how to memorize a deck of cards. I think this is a good exercise to execute deep concentration in small chunks of time, that is actually the most probably outcome in (most people) normal day.
You can do it. But you need to work hard for it. The society, working environment and yourself are not going to make it easy.
I think with the lockdown period, it is a good moment to put these techniques in practice.
I need to pay less attention to the emails and slack. I dont have to be the quickest answering something… (that is so good for your ego….) I need to really prioritize my working hours and tasks to focus on with a time frame. I need to make myself accountable, stop blaming somebody else. And communicate, make my peers that I will focus in things I will not answer immediately (if you are my CTO, maybe not 🙂 And as a manager, make my team members better: make them to take more ownership so they can deal with problems by themselves. And schedule times of the day to check emails and/or attend meetings.
One thing I have done, it is to put a pink sticker close to my screen saying “ONE THING AT EACH TIME”. I did this before reading the book as a reminder from a speech at work of a brilliant guy in his last day. And he said he learned that sentence from our CTO. That got burned in my mind. I have used it mainly from troubleshooting. It has been a critical tool that I have applied successfully many times since then. But I can be applied to more things as “Deep Work”
I have remembering my Karate for nearly the last two months. It has been a quite satisfying choice and has brought some good and old feelings.
As I have been adding katas to my set, I wanted to write and find some info in the web about the origin of Karate (and see if it matches my memory) and the main kata stiles I learned in my time. Not sure if it is still the same though.
I think for my black belt exam, apart from performing some katas in front of my teacher, I had to answer some questions about Karate history.
I dont know why, I can still remember bodhidharma, and Indian Buddhist monk, as the person considered for starting martial arts in Asia. Then it spread to southern China and then to Okinawa. There, it developed while some King forbade weapons so people needed other ways for self-defence. And finally, get to Japan after some conquering. Yeah, very short summary. Surely a better version here.
From my time, we had three stiles of “superior” katas: shuri-te, naha-te and tomari-te. I can’t forget the big picture of Gichin Funakoshi (Father of modern karate) in our tatami.
We used to consider naha-te katas the ones with a lot of “breathing” and “slow moves”. Katas with short and quick moves, were tomari-te katas.
And by chance, I was lucky to visit Okinawa (just for a weekend) when I was working in a project in Tokyo. It was a dream come true. Although I was expecting some spirituality there, I was lucky to attend a very important festival and laugh when some towns in the map where actually kata names!
Definitely, it is a very different place compared with Japan main islands.
I have finished reading this book about Kaizen. Many years ago I heard the term Kaizen for the superior productivity in Japan, mainly from Toyota as the world’s number one car producer. Somehow, I bought this used copy to learn about it.
First, I noticed the book was printed in 1986… I realised I rarely read “technical” stuff so “old”.
The first surprise was that it seems the concept of quality control was actually brought by USA to post WWII Japan. The two main people were W E Deming and J M Juran.
It is interesting that Japan was very eager to learn the productivity secrets from USA and at the end, they created their on version.
I like the focus in people. They need to be engaged and feel part of something. At the end of the day, everybody has to push together to get to great results. As well, it seems key the achievement of small changes and not massive ones. They set for long-term goals, mainly for customer satisfaction that is not just the person who buys the product. It is not all about profits. It seems the profits will come as a by-product (reduce cost, increase customer satisfaction, more sells -> more profits). So the vision is product-oriented instead of result-oriented.
The point that “if you dont have problems, how you can improve?!” is so true.
And one slogan to measure how good is your product: “would you buy what you are producing?”
I can see many concepts are already in place in technology. The “Kambam” board, the constant search for small improvements, etc. If you think devops culture is something really modern, doesn’t look like that.
In general, the approach is quite different from the Western world and has been successful. But the book mentions that you need the innovation side for keep improving. So again, as life, you need balance.
And at the end, Kaizen becomes like a way of life. Or it is like I see it.
I am curious how the author would see Kaizen and Japan nowadays.
This year, in my employer, I completed the migration to a MPLS SR Arista core network from a Brocade MPLS LDP one. Our backbone is still pure IPv4 so anything IPv6 is not going to be added. But this week, via an APNIC blog post I read about SRv6. And it looks quite interesting. So I went to the first post to go a bit deeper about what SRv6 is. Based on the statements of the blog, really big networks are already using this technology and quite a lot of support from the open source community too. I missed Arista in that list though.
So I tried to find some “real” proof of this SRv6 is some pcap files to see the format and get a bit better view. I could find at lest a source with some. The examples are not like the ones mentioned in the APNIC blog post but just for taking a look, it is enough:
So I can see inside the IPv6 header, the SRv6 Header as defined in the rfc.
I dont really understand the second IPv6 header (Dst: b::2). From the first IPv6 header, the destination “f1::” has to be the first instruction SID1. I can see how it mentions it contains a SRH (Next Header: 43). And inside the routing header, we can see it is SR type (Type: 4). I assume that Address[0] and Address[1] are SID2 and SID3.
I think the only thing I miss from Microsoft Windows, it is “paint”. I can’t find anything in Linux that is simple and reliable. I have used pinta, gimp and others but they break or they are too pro.
So, if you have internet connection. This is your friend
And if you want something a bit more pro like Visio, then this one
I recommend both a lot!
I know you have GIMP, but I just a simple tool to draw red squares!
Today I have had “fun” troubleshooting an issue that looked easy at first sight. A colleague was trying to PXE boot some server from a network that we haven’t used for a while.
When the server boots up, asks for an IP via DHCP. As we have a centralized DHCP server infrastructure, we have configured DHCP relay in the firewall facing that server to send that request to the DHCP server.
First, let’s take a look at how DHCP relay works. This is a very good link. And this diagram from the mentioned link it is really useful:
One think I learned is the reply (DCHP Offer) doesnt have to use as destination IP the same IP it received as source in DHCP Discover. In the picture, it is packet 2a.
Checking in our environment, we confirm that:
Our server is in 10.94.240.x network. Our firewall is acting as DHCP relay, and send the DHCP Discovery (unicast) to our VIP DHCP Server IP.
The DHCP offer, uses as source the physical IP of the DHCP server and destination is the DHCP relay IP (so it is 10.94.240.1 – the firewall IP in 10.94.240.x network)
Ok, so everything looks fine? No really. The server receives the query, it answers… but we dont see a DCHP Request/ACK.
BTW, keep in mind that DHCP is UDP….
So, we need to see where the packets are lost.
This is a high level path flow between the client and server:
So we need to check this connection is three different firewall vendors….
The initial troubleshooting was just using the GUI tools from Palo/Fortigate. We couldn see anything…. but the server was constantly receiving DHCP Discover and sending DHCP Offer… I dont get it:
# tcpdump -i X udp port 67 or 66 -nn
14:58:06.969462 IP 10.81.25.1.67 > 10.81.251.47.67: BOOTP/DHCP, Request from 6c:2b:59:c1:32:73, length 300
14:58:06.969564 IP 10.81.251.201.67 > 10.94.240.1.67: BOOTP/DHCP, Reply, length 300
14:58:28.329048 IP 10.81.25.1.67 > 10.81.251.47.67: BOOTP/DHCP, Request from 6c:2b:59:c1:32:73, length 300
14:58:28.329157 IP 10.81.251.201.67 > 10.94.240.1.67: BOOTP/DHCP, Reply, length 300
Initially it took me a while to see the request/reply because I was assuming the dhcp request had source 10.94.240.1. So I was seeing only the Reply but not the Request. That was when I went to clarify my head about DHCP Relay and found the link.
So ok, we have the DHCP Request/Reply, but absolutely nothing in the Palo. Is the palo dropping the packets or is forwarding? No idea. The GUI says nothing, I took a packet capture and couldnt see that traffic neither…
Doesnt makes sense.
Let’s get back to basic.
Did I mention DHCP is UDP? So how a next generation firewall (like paloalto) with all the fancy features enable (we have nearly all of them enable…) treats a UDP connection? UDP is stateless… but the firewall is statefull… the firewall creates a flow with the first packet so it can track, any new packet is considered part of that flow. But why we dont see the flows? We actually have only one flow. The firewall has created that session and offloaded to hardware. So you dont see anything else in the control-plane / GUI. The GUI only shows the end of a connection/flow. And as our flow DHCP Relay hasnt’ terminated (it is UDP) and the firewall keeps receiving packets, it is considered life (the firewall doesnt really know what is going on). So for that reason we dont see the connection in the PaloUI. Ok, I got to that point after a while…. I need to proof that the packet from the server is reaching the firewall and it is leaving it too.
How can I do that? Well, I need to delete that flow so the firewall considers a new connection and the tcpdump can see the packets.
This is the a good link from paloalto to take captures. So I found my connection and the cleared it:
palo(active)> show session all filter destination 10.94.240.1
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
135493 dhcp ACTIVE FLOW 10.81.251.201[67]/ZONE1/17 (10.81.251.201[67])
vsys1 10.94.240.1[67]/ZONE2 (10.94.240.1[67])
palo(active)>
palo(active)> clear session id 135493
And now, my packet capture in paloalto confirms that it is sending the packet to the next firewall (checking the destination MAC) !!!
Ok, so we confirm the first firewall in the return path was fine…. next one, it is fortigate.
BTW, we were checked and assumed that the routing is fine in all routers, firewalls, etc. Sometimes is not the case… so when things dont follow your thoughts, get back to the very basics….
We have exactly the same issue as in PaloAlto. I can’t see anything in the logs about receiving a dhcp offer from palo and forwarding it to the last firewall Cisco.
And again, we apply the same reasoning. We have an UDP connection, we have a next-generation firewall (with fancy ASIC). And one more thing, in this fortigate firewall, we allow intra-zone traffic, so it is not going to show anyway in the GUI monitor…
Fantastic, we have confirmation that the second firewall receives and forwards the DHCP Reply!!!
Ok, now the last stop, Cisco ASA. This is an old firewall, I think it could be my father or Darth Vader.
I dont have the fancy tools for packet capture like Palo/Fortigate…. so I went to the basic “debug” commands and “packet-tracer”.
First, this was the dhcp config in Cisco:
vader/pri/act# show run | i dhcp
dhcprelay server 10.81.251.47 EGRESS
dhcprelay enable SERVERS-ZONE
dhcprelay timeout 60
And, the ACL allows all IP traffic in those interfaces… and couldnt see any deny in the logs.
So, I enabled all debugging things I could find for dhcp:
vader/pri/act# show debug
debug dhcpc detail enabled at level 1
debug dhcpc error enabled at level 1
debug dhcpc packet enabled at level 1
debug dhcpd packet enabled at level 1
debug dhcpd event enabled at level 1
debug dhcpd ddns enabled at level 1
debug dhcprelay error enabled at level 1
debug dhcprelay packet enabled at level 1
debug dhcprelay event enabled at level 200
vader/pri/act# DHCPD: Relay msg received, fip=ANY, fport=0 on SERVERS-ZONE interface
DHCPRA: relay binding found for client f48e.38c7.1b6e.
DHCPD: setting giaddr to 10.94.240.1.
dhcpd_forward_request: request from f48e.38c7.1b6e forwarded to 10.81.251.47.
DHCPD: Relay msg received, fip=ANY, fport=0 on SERVERS-ZONE interface
DHCPRA: relay binding found for client 6c2b.59c1.3273.
DHCPD: setting giaddr to 10.94.240.1.
dhcpd_forward_request: request from 6c2b.59c1.3273 forwarded to 10.81.251.47.
vader/pri/act#
So, the debugging doesnt says anything regarding the packet coming back from Fortigate… Not looking good I am afraid. I wasnt running out of ideas about debug commands. I coudn’t increase an log level neither….
Let’s give a go to packet tracer… doesnt looks good:
vader/pri/act# packet-tracer input EGRESS udp 10.81.251.201 67 10.94.240.1 67
Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Result:
input-interface: EGRESS
input-status: up
input-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
So, we are sure our ACL is totally open but the firewall is dropping the packet coming from fortigate. Why? How to fix it?
Ok, get back to basics. Focus in Cisco config. It uses as DHCP relay server, 10.81.251.47 (VIP). But the DHCP reply is coming from the physical IP 10.81.251.201….. maybe Cisco doesnt like that…. Let’s try to add the physical IPs as a new DHCP server:
vader/pri/act# sri dhcp
dhcprelay server 10.81.251.47 EGRESS
dhcprelay server 10.81.251.201 EGRESS
dhcprelay server 10.81.251.202 EGRESS
I think I nearly cried after seeing this in the dhcp logs in our server:
May 12 16:16:27 dhcp1 dhcpd[2561]: DHCPDISCOVER from f4:8e:38:c7:1b:6e via 10.94.240.1
May 12 16:16:28 dhcp1 dhcpd[2561]: DHCPOFFER on 10.94.240.50 to f4:8e:38:c7:1b:6e (cmc-111) via 10.94.240.1
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 0 class decls to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 0 deleted host decls to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 0 new dynamic host decls to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: Wrote 1 leases to leases file.
May 12 16:16:28 dhcp1 dhcpd[2561]: DHCPREQUEST for 10.94.240.50 (10.81.251.202) from f4:8e:38:c7:1b:6e (cmc-111) via 10.94.240.1
May 12 16:16:28 dhcp1 dhcpd[2561]: DHCPACK on 10.94.240.50 to f4:8e:38:c7:1b:6e (cmc-111) via 10.94.240.1
So at the end, finally fixed…. it took too many hours.
Notes:
DHCP Realy: It is not that obvious the flow regarding IPs.
UDP and firewalls, debugging it is a bit more challenging.
Cisco ASA dhcprelay server IPs…. VIPs and non-VIPs please.
Another issue I had during the weekend that took me hours. Thanks that I have been reading a bit this book (1.6) and had some clues.
I was trying to test a repo to start an Arista lab using docker and I assumed that everything should work if I followed the instructions. My problem was the script trying to push some basic config to the switches.
As you can see that differs with what we have inside the bash script. So from the bash script we need to put between ‘ the parameter for -c but inside the parameter we need to use “. So I had to make the change below:
- docker exec "${1}" bash -c "echo -e ${commands} | FastCli -p15 -e"
+ # need to update this command as the quoting doesnt work in my bash
+ docker exec "${1}" bash -c 'echo -e '"'${commands}'"' | FastCli -p15 -e'
The books says Enclose a string in single quotes ‘ unless it contains elements that you want the shell to interpolate. So let’s divide the solution in parts so can be easier to digest (and remember for me in the future because this will bit me again for sure)
1st part: ‘echo -e ‘
2nd part: “
3rd part: ‘${commands}’
4th part: “
5th part: ‘ | FastCli -p15 -e’
The ” need to be outside the ‘ region because the commands need to be between ” for the docker command. The 3rd part will expand the variable commands.
I guess the author is using a different version of bash? This is mine
$ bash --version
GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software; you are free to change and redistribute it.
Something very basic but took me several hours to workout. I had a VM that I wanted to increase a VG as I wanted to create a new LV. I increased the partition in the host server so the PV of the VG had the extra space, but then I couldnt see the increase inside the VM:
"fdisk" was telling me the disk was already 300G...
[root@VM ~]# fdisk /dev/vdb
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
The old LVM2_member signature will be removed by a write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xd46fa2fc.
Command (m for help): p
Disk /dev/vdb: 300 GiB, 322122547200 bytes, 629145600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd46fa2fc
I did a pvscan... and nothing. What I was missing? just "pvresize".... and then I can see my extra 100G in the PV and in the VG. So I can create the new LV I wanted...
