OOB

I was reading this blog and realised that OOB is something is not talked about very often. Based on what I have seen in my career:

Design

You need to sell the idea that this is a must. Then you need to secure some budget. You dont need much:

1x switch

1x firewall

1x Internet access (if you have your ASN and IP range, dont use it)

Keep it simple..

Most network kit (firewalls, routers, switches, pdus, console servers, etc) have 1xmgmt port and 1xconsole port. So all those need to go to the console server. I guess most server vendors offer some OOB access (I just know Dell and HP). So all those go to the oob switch.

If you have a massive network with hundreds of devices/servers, then you will need more oob switches and console servers. You still need just one firewall and 1 internet connection. The blog comments about the spine-leaf oob network. I guess this is the way for a massive network/DC.

Access to OOB

You need to be able to access it via your corporate network and from anywhere in the internet.

You need to be sure linux/windows/macs can VPN.

Use very strong passwords and keys.

You need to be sure the oob firewall is quite tight in access. At the end of the day you only want to allow ssh to the console server and https to the ILO/iDRACS. Nothing initiated internally can go to the internet.

Dependencies

Think in the worse scenario. Your DNS server is down. Your authentication is down.

You need to be sure you have local auth enabled in all devices for emergency

You need to work out some DNS service. Write the key IPs in the documentation?

You IP transit has to be reliable. You dont need a massive pipe but you need to be sure it is up.

Monitoring

You dont want to be in the middle of the outage and realise that your OOB is not functional. You need to be sure the ISP for the OOB is up and the devices (oob switch and oob firewall) are functional all the time.

How to check the serial connections? conserver.com

Documentation

Another point frequently lost. You need to be sure people can find info about the OOB: how is built and how to access it.

Training

At the end of the day, if you have a super OOB network but then nobody knows how to connect and use it, then it is useful. Schedule routine checkups with the team to be sure everybody can OOB. This is useful when you get a call at 3am.

Diagram

Update

Funny enough, I was watching today NLNOG live and there was a presentation about OOB with too different approaches: in-band out-of-band and pure out-of-band.

From the NTT side, I liked the comment about conserver.com to manage your serial connections. I will try to use it once I have access to a new network.

Forward TCPDump to Wireshark

Reading this blog entry I realised that very likely I have never tried forward tcpdump to a wireshark. How many times I have taken a pcap in a switch and then I need to download to see the details in wireshark…

I guess you can find some blocking points in firewalls (at least for 2-steps option)

So I tried the single command with a switch in my ceoslab and it works!

Why it works?

ssh <username>@<switch>  "bash tcpdump -s 0 -U -n -w - -i <interface>" | wireshark -k -i -

The ssh command is actually executing the “bash tcpdump…” remotely. But the key is the “-U” and “-w -” flags. “-U” in conjunction with “-w” sends the packet without waiting for the buffer to fill. Then “-w -” says that it writes the output to stdout instead of a file. If you run the command without -U, it would work but it will update a bit slower as it needs to fill the buffers.

From tcpdump manual:

       -U
       --packet-buffered
              If the -w option is not specified, make the printed packet output ``packet-buffered''; i.e., as the description of the contents of each packet is printed, it will be written to the standard  output,  rather  than, when not writing to a terminal, being written only when the output buffer fills.

              If  the  -w  option  is  specified, make the saved raw packet output ``packet-buffered''; i.e., as each packet is saved, it will be written to the output file, rather than being written only when the output buffer fills.

              The -U flag will not be supported if tcpdump was built with an older version of libpcap that lacks the pcap_dump_flush() function.

......
   -w file
          Write the raw packets to file rather than parsing and printing them out.  They can later be printed with the -r option.  Standard output is used if file is ``-''.

          This  output will be buffered if written to a file or pipe, so a program reading from the file or pipe may not see packets for an arbitrary amount of time after they are received.  Use the -U flag to cause packets to be written as soon as they are received.

And the stdout of that process is the ssh command so we redirect that outout via a pipe “|” and it is sent as input for wireshark thanks to “-i -” that makes wireshark to read from stdin (that is the stdout from the tcpdump in the switch!)

The wireshark manual:

       -i|--interface  <capture interface>|-
           Set the name of the network interface or pipe to use for live packet capture.

           Network interface names should match one of the names listed in "wireshark -D" (described above); a number, as reported by "wireshark -D", can also be used.  If you're using UNIX, "netstat -i", "ifconfig -a" or "ip link" might also work to list interface names, although not all versions of UNIX support the -a flag to ifconfig.

           If no interface is specified, Wireshark searches the list of interfaces, choosing the first non-loopback interface if there are any non-loopback interfaces, and choosing the first loopback interface if there are no non-loopback interfaces.  If there are no interfaces at all, Wireshark reports an error and doesn't start the capture.

           Pipe names should be either the name of a FIFO (named pipe) or "-" to read data from the standard input.  On Windows systems, pipe names must be of the form "\\pipe\.\pipename".  Data read from pipes must be in standard pcapng or pcap format. Pcapng data must have the same endianness as the capturing host.

           This option can occur multiple times. When capturing from multiple interfaces, the capture file will be saved in pcapng format.

....

       -k  Start the capture session immediately.  If the -i flag was specified, the capture uses the specified interface.  Otherwise, Wireshark searches the list of interfaces, choosing the first non-loopback interface if
           there are any non-loopback interfaces, and choosing the first loopback interface if there are no non-loopback interfaces; if there are no interfaces, Wireshark reports an error and doesn't start the capture.

The two-steps option relies on “nc” to send/receive the data, but it is the same idea regarding the tcpdump/wireshark flags using “-“

On switch: tcpdump -s 0 -U -n -w - -i <interface> | nc <computer-ip> <port>
On PC: netcat -l -p <port> | wireshark -k -S -i -

Linux Networking – Bonding/Bridging/VxLAN

Bonding

$ sudo modprobe bonding
$ ip link help bond
$ sudo ip link add bond0 type bond mode 802.3ad
$ sudo ip link set eth0 master bond0
$ sudo ip link set eth1 master bond0

Bridging: vlans + trunks

ip neigh show // l2 table
ip route show // l3 table

ip route add default via 192.168.1.1 dev eth1

sudo modprobe 8021q

// create bridge and add links to bridge (switch)
sudo ip link add br0 type bridge vlan_filtering 1 // native vlan = 1
sudo ip link set eth1 master br0
sudo ip link set eth2 master br0
sudo ip link set eth3 master br0

// make eth1 access port for v11
sudo bridge vlan add dev eth1 vid 11 pvid untagged

// make eth3 access port for v12
sudo bridge vlan add dev eth3 vid 12 pvid untagged

// make eth2 trunk port for v11 and v12
sudo bridge vlan add dev eth2 vid 11
sudo bridge vlan add dev eth2 vid 12

// enable bridge and links
sudo ip link set up dev br0
sudo ip link set up dev eth1
sudo ip link set up dev eth2
sudo ip link set up dev eth3

bridge link show
bridge vlan show
bridge fdb show

VxLAN

I havent tried this yet:

Linux System 1
  sudo ip link add br0 type bridge vlan_filtering 1
  sudo ip link add vlan10 type vlan id 10 link bridge protocol none
  sudo ip addr add 10.0.0.1/24 dev vlan10
  sudo ip link add vtep10 type vxlan id 1010 local 10.1.0.1 remote 10.3.0.1 learning
  sudo ip link set eth1 master br0
  sudo bridge vlan add dev eth1 vid 10 pvid untagged

Linux System 2
  sudo ip link add br0 type bridge vlan_filtering 1
  sudo ip link add vlan10 type vlan id 10 link bridge protocol none
  sudo ip addr add 10.0.0.2/24 dev vlan10
  sudo ip link add vtep10 type vxlan id 1010 local 10.3.0.1 remote 10.1.0.1 learning
  sudo ip link set eth1 master br0
  sudo bridge vlan add dev eth1 vid 10 pvid untagged

Traceroute

A good refresh about traceroute. It is a very common tool for network troubleshooting so it is important to use it wisely

Important points

  • ICMP vs UDP: most implementations do UDP (it can be blocked…)
  • Every probe is an independent trial!
  • Try to identify the characteristics and location of each hop
  • If there is a congestion/delay issue in one hop, it has to be carried out to the next hops, if not, it is just prioritizing of the ICMP generation by that router/hop.
  • You dont see the reverse path – Ask the other end (if possible) to send the traceroute from its end.
  • Border routers between providers can be a hot spot for issues.
  • Asymmetric paths can bite you. Try to set the source address in your tests (from the provider IP, from your own space, etc)
  • Spot ECMP (in the same hop, you see several different IPs). Multiple unequal length paths can be painful.
  • MPLS: most times is hidden (TTL is removed). It can be tricky to spot. But it can be funny when you see the hops (with private IPs 🙂

And if you are more interested in the paths than latency, this can be a good too:

https://github.com/rucarrol/traceflow

Openconfig

I have struggled to get something working for learning a bit of openconfig.

At the end, all info my info comes from Anton’s Karneliuk blog series about Openconfig. So all credit to him. It is the best source about real testing of openconfig in different platforms.

I am not going to create the wheel and explain what openconfig is. In my head, it is attempt from several big vendors to standardise the network management (config) and monitoring (telemetry) via YANG (vendor-neutral) models. So OC uses YANG. And we interact with OC using a transport protocol like netconf, restcong and gNMI. So the network devices need to implement one of these protocols. Based on the blog Cisco, Nokia and Arista have netconf implementations and Ansible has a module for that!!! So the key words are openconfig, yang and netcong.

So in my case, based on my ceos lab, I have added a new playbook based on Anton’s to test openconfig/netcong with Arista cEOS:

https://github.com/thomarite/ceos-testing/blob/master/README.md#openconfig

This is quite basic as it only gets the interface config.

Following Anton’s Part3 Blog:

I tried to push config via openconfig to my ceos devices (all files are in github as per link above).

The blog is dense but it is good because there is a lot of info. In this case, you have to use an Ansible role so it is a new thing to learn. As well, I wanted to adapt that role to my env and found some ansible issues but managed to fix after reading ansible documentation and paying attention to the -vvv info.

From “oc-push-config.yaml” the first task “collect” it is fine. It just takes some 10 minutes or more to get all YANG modules from each device.

The issue is with task “configure”. It fails when trying to push the interface config. I have tried Anton’s config and the actual config generated from oc-get-interface-info.yaml but no joy.

Based on the blog, it seems Arista doesnt have much interest in Openconfig.

Anyway, there have been a couple of intense days looking at all this openconfig/netcong/yang thing. I have just touched the surface but I have learned some Ansible in the way too. So could be worse.

Will move on to something else.

Netbox – API Troubleshooting

Yesterday managed to get netbox and my lab connected. So today followed up with the original article, and found a new issue that took me several hours.

Initially I was seeing an error that I couldn’t undestand “

netbox.exceptions.CreateException: This field is required

From

(venv) /netbox-example/nornir-napalm-netbox-demo master$ python scripts/create_interfaces.py
nb_url = http://0.0.0.0:8080
Creating Netbox Interface for device r1, interface Loopback1
Traceback (most recent call last):
File "scripts/create_interfaces.py", line 42, in
task=create_netbox_interface,
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/init.py", line 146, in run
result = self._run_serial(task, run_on, **kwargs)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/init.py", line 72, in _run_serial
result[host.name] = task.copy().start(host, self)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/task.py", line 85, in start
r = self.task(self, **self.params)
File "scripts/create_interfaces.py", line 34, in create_netbox_interface
device_id=device_id,
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/dcim.py", line 431, in create_interface
return self.netbox_con.post('/dcim/interfaces/', required_fields, **kwargs)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/connection.py", line 124, in post
raise exceptions.CreateException(resp_data)
netbox.exceptions.CreateException: This field is required.

So I started to follow the trace, adding “print” and using “ipdb” to see what was going on:

....
/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/connection.py(71)__request()
70 finally:
---> 71 self.close()
72
ipdb> dir(response)
['attrs', 'bool', 'class', 'delattr', 'dict', 'dir', 'doc', 'enter', 'eq', 'exit', 'format', 'ge', 'getattribute', 'getstate', 'gt', 'hash', 'init', 'init_subclass', 'iter', 'le', 'lt', 'module', 'ne', 'new', 'nonzero', 'reduce', 'reduce_ex', 'repr', 'setattr', 'setstate', 'sizeof', 'str', 'subclasshook', 'weakref', '_content', '_content_consumed', '_next', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'next', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']
ipdb> response.url
'http://0.0.0.0:8080/api/dcim/interfaces/'
ipdb> response.text
'{"type":["This field is required."]}'
ipdb> response.status_code
400
ipdb> response.content
b'{"type":["This field is required."]}'
ipdb> response.reason
'Bad Request'
ipdb> response.request

ipdb> prepared_request

ipdb> prepared_request.url
'http://0.0.0.0:8080/api/dcim/interfaces/'
ipdb> dir(prepared_request)
['class', 'delattr', 'dict', 'dir', 'doc', 'eq', 'format', 'ge', 'getattribute', 'gt', 'hash', 'init', 'init_subclass', 'le', 'lt', 'module', 'ne', 'new', 'reduce', 'reduce_ex', 'repr', 'setattr', 'sizeof', 'str', 'subclasshook', 'weakref', '_body_position', '_cookies', '_encode_files', '_encode_params', '_get_idna_encoded_host', 'body', 'copy', 'deregister_hook', 'headers', 'hooks', 'method', 'path_url', 'prepare', 'prepare_auth', 'prepare_body', 'prepare_content_length', 'prepare_cookies', 'prepare_headers', 'prepare_hooks', 'prepare_method', 'prepare_url', 'register_hook', 'url']
ipdb> prepared_request.path_url
'/api/dcim/interfaces/'
ipdb> response.__content
*** AttributeError: 'Response' object has no attribute '__content'
ipdb> response._content
b'{"type":["This field is required."]}'
ipdb> response.content
b'{"type":["This field is required."]}'
ipdb> response.headers
{'Server': 'nginx', 'Date': 'Wed, 08 Jul 2020 12:36:35 GMT', 'Content-Type': 'application/json', 'Content-Length': '36', 'Connection': 'keep-alive', 'Vary': 'Accept, Cookie, Origin', 'Allow': 'GET, POST, HEAD, OPTIONS, TRACE', 'API-Version': '2.8', 'X-Content-Type-Options': 'nosniff', 'X-Frame-Options': 'SAMEORIGIN'}
ipdb> response.reason
'Bad Request'
ipdb> response.request

ipdb> response.test
*** AttributeError: 'Response' object has no attribute 'test'
ipdb> response.text
'{"type":["This field is required."]}'
ipdb> response.url
'http://0.0.0.0:8080/api/dcim/interfaces/'
ipdb> quit
Create Netbox Interfaces
r1 ** changed : False
vvvv Create Netbox Interfaces ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ERROR
---- napalm_get ** changed : False --------------------------------------------- INFO
(venv) go:1.12.5|py:3.7.3|tomas@athens:~/storage/technology/netbox-example/nornir-napalm-netbox-demo master$ python scripts/create_interfaces.py
nb_url = http://0.0.0.0:8080
url3=http://0.0.0.0:8080/api/dcim/interfaces?limit=0
Creating Netbox Interface for device r1, interface Loopback1
url3=http://0.0.0.0:8080/api/dcim/devices/?name=r1&limit=0
device_id = 1
url3=http://0.0.0.0:8080/api/dcim/interfaces/
resp_ok=False resp_status=400
body_data= {'name': 'Loopback1', 'form_factor': 1200, 'device': 1}
params= /dcim/interfaces/
resp_data= {'type': ['This field is required.']}
Traceback (most recent call last):
File "scripts/create_interfaces.py", line 43, in
task=create_netbox_interface,
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/init.py", line 146, in run
result = self._run_serial(task, run_on, **kwargs)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/init.py", line 72, in _run_serial
result[host.name] = task.copy().start(host, self)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/task.py", line 85, in start
r = self.task(self, **self.params)
File "scripts/create_interfaces.py", line 35, in create_netbox_interface
device_id=device_id,
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/dcim.py", line 431, in create_interface
return self.netbox_con.post('/dcim/interfaces/', required_fields, **kwargs)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/connection.py", line 130, in post
raise exceptions.CreateException(resp_data)
netbox.exceptions.CreateException: This field is required.

So it seems that at the end I realised that I was missing the parameter “type” !!!

I was checking the documentation from netbox in github but I couldnt see clearly what kind of config I had to provide…

I checked the “type” value for the only interfaces I already had in netbox: “http://0.0.0.0:8080/api/dcim/interfaces/

So I tried to pass exactly that but it was still failing…

(venv) go:1.12.5|py:3.7.3|tomas@athens:~/storage/technology/netbox-example/nornir-napalm-netbox-demo master$ python scripts/create_interfaces.py
nb_url = http://0.0.0.0:8080
url3=http://0.0.0.0:8080/api/dcim/interfaces?limit=0
Creating Netbox Interface for device r1, interface Loopback1
url3=http://0.0.0.0:8080/api/dcim/devices/?name=r1&limit=0
device_id = 1
url3=http://0.0.0.0:8080/api/dcim/interfaces/
resp_ok=False resp_status=400
body_data= {'name': 'Loopback1', 'form_factor': 1200, 'device': 1, 'type': {'value': '1000base-t', 'label': '1000BASE-T (1GE)', 'id': 1000}}
params= /dcim/interfaces/
resp_data= {'type': ['Value must be passed directly (e.g. "foo": 123); do not use a dictionary or list.']}
Traceback (most recent call last):
File "scripts/create_interfaces.py", line 50, in
task=create_netbox_interface,
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/init.py", line 146, in run
result = self._run_serial(task, run_on, **kwargs)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/init.py", line 72, in _run_serial
result[host.name] = task.copy().start(host, self)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/nornir/core/task.py", line 85, in start
r = self.task(self, **self.params)
File "scripts/create_interfaces.py", line 42, in create_netbox_interface
**interface_type,
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/dcim.py", line 431, in create_interface
return self.netbox_con.post('/dcim/interfaces/', required_fields, **kwargs)
File "/home/tomas/storage/technology/netbox-example/venv/lib/python3.7/site-packages/netbox/connection.py", line 130, in post
raise exceptions.CreateException(resp_data)
netbox.exceptions.CreateException: Value must be passed directly (e.g. "foo": 123); do not use a dictionary or list.
(venv) go:1.12.5|py:3.7.3|tomas@athens:~/storage/technology/netbox-example/nornir-napalm-netbox-demo master$

Somehow the API had to be documented… by chance, looking at the bottom of the netbox page, there was an”API” link….

So, now I needed to look up the correct API call. Based on the script and logs, it was a “POST” for “/dcim/interfaces/”. Here we go!

So finally, I had the info. I confirmed what fields were mandatory and the value they needed!

interface_type = {}
interface_type["type"] = "1000base-t"
for interface_name in interfaces.keys():
    if not is_interface_present(nb_interfaces, f"{task.host}", interface_name):
        print(
            f"* Creating Netbox Interface for device {task.host}, interface {interface_name}"
        )
        device_id = get_device_id(f"{task.host}", netbox)
        print("device_id = %s" % device_id)
        netbox.dcim.create_interface(
           name=f"{interface_name}",
           form_factor=1200,  # default
           device_id=device_id,
           **interface_type,
        )

So the script ran fine for all my devices:

netbox-example/nornir-napalm-netbox-demo master$ python scripts/create_interfaces.py
nb_url = http://0.0.0.0:8080
url3=http://0.0.0.0:8080/api/dcim/interfaces?limit=0
Create Netbox Interfaces
r1 ** changed : False
vvvv Create Netbox Interfaces ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_get ** changed : False --------------------------------------------- INFO
^^^^ END Create Netbox Interfaces ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
r2 ** changed : False
vvvv Create Netbox Interfaces ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_get ** changed : False --------------------------------------------- INFO
^^^^ END Create Netbox Interfaces ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
r3 ** changed : False
vvvv Create Netbox Interfaces ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_get ** changed : False --------------------------------------------- INFO
^^^^ END Create Netbox Interfaces ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

And it is updated in GUI:

Netbox

Another thing I wanted to play with is Netbox and found a good article to follow. So credits to the authors.

Using my current ceos lab from https://github.com/thomarite/ceos-testing

I followed Rick’s article to install netbox-docker and his own repo with the nornir examples using netbox. In this case nornir is going to use netbox as inventory. Normally I use local files. I created a python venv for 3.7.3

mkdir netbox-example; cd netbox-example
pyenv local 3.7.3
python -m virtualenv venv
source venv/bin/activate
git clone https://github.com/netbox-community/netbox-docker.git
cd netbox-docker
vim docker-compose.yml  --> so it always expose 8080
      nginx:
      ...
        ports:
          - 8080:8080
docker-compose pull
docker-compose up

When installing the requirements for “nornir-napalm-netbox-demo” I had to modify the version of some packages. So I removed the required version and I left pip to install the latest. I didnt use the makefile.

git clone https://github.com/rickdonato/nornir-napalm-netbox-demo
cd nornir-napalm-netbox-demo
python -m pip install -r requirements.txt

I struggled quite a bit with the management IP in netbox and the meaning of “platform”

  • Create Manufacturers under Device Types: I created “Arista”
  • Create Device Types under Device Types: I created “ceos”
  • Create Platforms under Devices: This is VERY important as it has to be a supported NAPALM platform!!! So for Arista, I need “eos”.
  • Create Device Roles under Devices. I created “pe”
  • Create Devices under Devices.
  • Within each device: add a management interface. Here, I got confused as I was adding the interface in the inventory section. The inventory section is info to/from the device using NAPALM. So you need to go to the bottom of the page, add the interface
  • and then add an IP to that interface and mark it as primary.

Keep in mind that initially, I was using “0.0.0.0” for each device as that’s the IP I have be using for all my scripts lately.

Keep in mind (II) that we are using docker twice (from different commands…) one to get netbox and the other via docker(-topo) to get the Arista ceos containers…. and we have iptables rules under the hood created by both…

But, let’s go step by step. Now we need to confirm that our nornir scrip can connect to netbox. So follow “Nornir-to-Netbox Configuration” section. This is my file. I updated the nb_url and nb_token. Notice the usage of “transform_function“.

---
core:
num_workers: 20
inventory:
plugin: nornir.plugins.inventory.netbox.NBInventory
options:
nb_url: 'http://0.0.0.0:8080'
nb_token: '<NETBOX_API_TOKEN>'
ssl_verify: False
transform_function: "helpers.adapt_user_password"

You need to update “scripts/secrets.py” with the devices you have in your inventory and the user/pass:

creds = {
"r1": {"username": "user", "password": "pas123"},
"r2": {"username": "user", "password": "pas123"},
"r3": {"username": "user", "password": "pas123"},
}

So now we can test if nornir can connect to netbox:

/netbox-example/nornir-napalm-netbox-demo master$ python scripts/helpers.py --inventory
{'defaults': {'connection_options': {},
'data': {},
'hostname': None,
'password': None,
'platform': None,
'port': None,
'username': None},
'groups': {},
'hosts': {'r1': {'connection_options': {},
'data': {'asset_tag': 'r1',
'model': 'ceos',
'role': 'pe',
'serial': 'r1',
'site': 'lab1',
'vendor': 'Arista'},
'groups': [],
'hostname': '192.168.16.2',
'password': 'pas123',
'platform': 'eos',
'port': None,
'username': 'user'},
'r2': {'connection_options': {},
'data': {'asset_tag': 'r2',
'model': 'ceos',
'role': 'pe',
'serial': 'r2',
'site': 'lab1',
'vendor': 'Arista'},
'groups': [],
'hostname': '192.168.16.3',
'password': 'pas123',
'platform': 'eos',
'port': None,
'username': 'user'},
'r3': {'connection_options': {},
'data': {'asset_tag': 'r3',
'model': 'ceos',
'role': 'pe',
'serial': 'r3',
'site': 'lab1',
'vendor': 'Arista'},
'groups': [],
'hostname': '192.168.16.4',
'password': 'pas123',
'platform': 'eos',
'port': None,
'username': 'user'}}}

All good. Let’s see if we can get backups from the devices.

netbox-example/nornir-napalm-netbox-demo master$ python scripts/backup_configs.py
Backup Device configurations**
r1 ** changed : True *
vvvv Backup Device configurations ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_get ** changed : False --------------------------------------------- INFO
---- write_file ** changed : True ---------------------------------------------- INFO
^^^^ END Backup Device configurations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
r2 ** changed : True *
vvvv Backup Device configurations ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_get ** changed : False --------------------------------------------- INFO
---- write_file ** changed : True ---------------------------------------------- INFO
^^^^ END Backup Device configurations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
r3 ** changed : True *
vvvv Backup Device configurations ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
---- napalm_get ** changed : False --------------------------------------------- INFO
---- write_file ** changed : True ---------------------------------------------- INFO
^^^^ END Backup Device configurations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(venv) /netbox-example/nornir-napalm-netbox-demo master$

Great all good.

Now, let’s see netbox using NAPALM. If you click on “Status” for any device, netbox will use NAPALM to get the facts from the device. If netbox is not configured properly with NAPALM, it will fail. This is a working scenario:

The tabs “LLDP neighbors” and “Configuration” relay too in NAPALM.

So for configuring netbox with napalm you need to tell netbox the user/pass that NAPALM needs:

netbox-example$ vim netbox-docker/env/netbox.env
...
NAPALM_USERNAME=user
NAPALM_PASSWORD=pas123
NAPALM_TIMEOUT=10
...

Very likely you will have to restart netbox:

/netbox-docker release$ docker-compose down
/netbox-docker release$ docker-compose up

As mentioned before, I had an issue when I was using “0.0.0.0” as IP. By default (as It seems I can’t think) I was using the exposed IP/port from docker-topo to reach the ceos switches. I haven’t had an issue until using netbox.

I am using docker for netbox and docker(-topo) for my arista cEOS switches. So the connectivity between netbox and ceos is via the IPs/interfaces/bridges created by docker. And remember… you have iptables under the hood. My first mistake was telling netbox to use 0.0.0.0 as it is the one I am using to testing from my scripts when connecting to ceos. Netbox needs to point to the IP assigned by docker :facepalm: 192.16.16.x in my case. Second one, the port, same thing docker exposes the port 443 as 900x for external connections and I use 900x in my scripts. From netbox point of view, it is still 443 :facepalm: And finally, I am calling docker twice for building my lab, one for netbox-docker and the other for ceos. You need to keep an eye on iptables changes when restarting netbox via docker-compose because you can be in the situation that netbox traffic is dropped in DOCKER-ISOLATION-STAGE-1 :facepalm: (need to try to write a docker-compose to build everything in one go)

So when I was having errors from netbox that it was being rejected when connecting to ceos devices via NAPALM, I couldnt understand it. My scripts were fine using those details (0.0.0.0:900x)

I ran tcpdump on one ceos on the “ethernet0” interface and NOTHING was hitting the interface from netbox on port 900x but my scripts could…..

Somehow netbox wasnt able to reach ceos r1??? In my head, netbox and ceos devices were all in 0.0.0.0….. so no routing, no firewalls, they are connected in the same network 0.0.0.0…..

At the end I waked up and realised that the docker devices are using the IPs provided by docker so it is following normal routing… and firewalling by iptables. The same for ceos devices, they have IPs (different from 0.0.0.0)

So I updated netbox with the correct management IPs for r1, r2 and r3 ceos.

When I filtered by the real netbox IP in r1 tcpdump ethernet0, I was seeing traffic on 900x!!! Good. Then I realised that it has to be 443. So I removed my hack to update the port to 900x.

For a different reason I had to restart docker-topo (for ceos) and then docker netbox. And now, I coudnt see any traffic from netbox in r1….. I “didn’t” change anything. So the routing didnt change, there was something else “cutting” the connection: iptables

docker uses iptables very heavily. I realised that after restart docker-netbox, iptables changed…

before restart:

# iptables -t filter -S DOCKER-ISOLATION-STAGE-1
Warning: iptables-legacy tables present, use iptables-legacy to see them
-N DOCKER-ISOLATION-STAGE-1
-A DOCKER-ISOLATION-STAGE-1 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-94a8183a4fb1 ! -o br-94a8183a4fb1 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-0d4ec9aba9bd ! -o br-0d4ec9aba9bd -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-609619313dc8 ! -o br-609619313dc8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-61d32350cb58 ! -o br-61d32350cb58 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-384488acbc99 ! -o br-384488acbc99 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN

after restart:

# iptables -t filter -S DOCKER-ISOLATION-STAGE-1
Warning: iptables-legacy tables present, use iptables-legacy to see them
-N DOCKER-ISOLATION-STAGE-1
-A DOCKER-ISOLATION-STAGE-1 -i br-381cdff63d2f ! -o br-381cdff63d2f -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-94a8183a4fb1 ! -o br-94a8183a4fb1 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-0d4ec9aba9bd ! -o br-0d4ec9aba9bd -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-609619313dc8 ! -o br-609619313dc8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-61d32350cb58 ! -o br-61d32350cb58 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN

With the restart, docker created a new bridge interface for netbox (old: br-384488acbc99, new: br-381cdff63d2f) and it wasnt hitting anymore the “DOCKER-ISOLATION-STAGE-1 -j ACCEPT”

So I had to make an iptables change:

# iptables -t filter -D DOCKER-ISOLATION-STAGE-1 -j ACCEPT
# iptables -t filter -I DOCKER-ISOLATION-STAGE-1 -j ACCEPT
# iptables -t filter -S DOCKER-ISOLATION-STAGE-1
Warning: iptables-legacy tables present, use iptables-legacy to see them
-N DOCKER-ISOLATION-STAGE-1
-A DOCKER-ISOLATION-STAGE-1 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-381cdff63d2f ! -o br-381cdff63d2f -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-94a8183a4fb1 ! -o br-94a8183a4fb1 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-0d4ec9aba9bd ! -o br-0d4ec9aba9bd -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-609619313dc8 ! -o br-609619313dc8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-61d32350cb58 ! -o br-61d32350cb58 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
#

And finally, netbox could use napalm to contact the ceos devices…. Calling docker twice is not a great idea….

BTW, this is my docker ps with netbox and ceos devices:

(venv) /netbox-example$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c23be76ffd54 nginx:1.17-alpine "nginx -c /etc/netbo…" 4 hours ago Up 4 hours 80/tcp, 0.0.0.0:8080->8080/tcp netbox-docker_nginx_1
5a0b89f18578 netboxcommunity/netbox:latest "/opt/netbox/docker-…" 4 hours ago Up 4 hours netbox-docker_netbox_1
528948de329b netboxcommunity/netbox:latest "python3 /opt/netbox…" 4 hours ago Up 4 hours netbox-docker_netbox-worker_1
29529302ba1c redis:5-alpine "docker-entrypoint.s…" 4 hours ago Up 4 hours 6379/tcp netbox-docker_redis_1
5e975ec2aa70 redis:5-alpine "docker-entrypoint.s…" 4 hours ago Up 4 hours 6379/tcp netbox-docker_redis-cache_1
6158672a4ae6 postgres:11-alpine "docker-entrypoint.s…" 4 hours ago Up 4 hours 5432/tcp netbox-docker_postgres_1
34841aa098d4 ceos-lab:4.23.3M "/sbin/init systemd.…" 5 hours ago Up 5 hours 0.0.0.0:2002->22/tcp, 0.0.0.0:9002->443/tcp 3node_r03
4ca92c6a3b09 ceos-lab:4.23.3M "/sbin/init systemd.…" 5 hours ago Up 5 hours 0.0.0.0:2001->22/tcp, 0.0.0.0:9001->443/tcp 3node_r02
67e8b7ab84e0 ceos-lab:4.23.3M "/sbin/init systemd.…" 5 hours ago Up 5 hours 0.0.0.0:2000->22/tcp, 0.0.0.0:9000->443/tcp 3node_r01

I was painful but I learned a couple of things about netbox, nornir and docker/iptables!!!

Nornir

Nornir is a python framework mainly for network automation. Instead of using another tool like Ansible (that you need to learn), you can do the same just using pure python all the way. Ansible doesnt scale well and can be very slow, with nornir you have threading from day zero, so if you have to run tasks in 100 devices, you will feel and see the difference.

I learnt about nornir via Kirk Byers’ course. Unfortunately I didnt have the chance/time to use it in my former day job so now I have had time to review things and do a small project.

From https://github.com/thomarite/ceos-testing in the nornir section you can find the whole environment. I tested on the 3-node topology.

It is nothing special. The script builds the config for BGP or ISIS using jinj2 and yaml files. I have the feeling that my jinja2 is a bit difficult to follow. Then using napalm connects to the devices to push or check the config.

Just one issue, as it seems due to the nature of cEOS relaying on docker and my filesystem, if you decide to push the config (dry_run=False == commit=True) the task will fail (while trying to write startup config) but it is actually executed.

(testdir2) /testdir2/ceos-testing/nornir master$ python buid-config.py -b isis -c
hostname: r1
task: deploy_config for isis
failed: True
logs: Traceback (most recent call last):
...
File ".../testdir2/lib/python3.7/site-packages/pyeapi/eapilib.py", line 469, in send
raise CommandError(code, msg, command_error=err, output=out)
pyeapi.eapilib.CommandError: Error [1000]: CLI command 5 of 5 'write memory' failed: could not run command [Error copying system:/running-config to flash:/startup-config (Operation not permitted)]
changed: False
diff:

hostname: r2
task: deploy_config for isis
failed: False
logs: None
changed: False
diff:

hostname: r3
task: deploy_config for isis
failed: False
logs: None
changed: False
diff:

This shouldn’t happen on vEOS or the real hardware (if you have the correct aaa config of course)

Route Reflectors – Notes

Reflect:
client -> clients and non-clients
non-client -> clients
No Reflect:
non-client -> no-client (normal ibgp)

always advertises to eBGP peers (normal ebgp)
eBGP learned prefixes, advertised to client and non-clients (normal ebgp)

Full Mesh iBGP:

  • between RRs
  • RRs and non-clients
  • clients just need iBGP to RRs.

Reflects ONLY the best route

RRs dont modify on reflected routes: NEXT_HOP, AS_PATH, LP and MED.

Prevent routing information loops: ORIGINATOR_ID and CLUSTER_LIST

Clustering:

  • ORIGINATOR_ID: The first RR creates the Originator_ID and sets it to the BGP router ID of the router that originates the route. So when a client receives a route with its own Originator ID, it is dropped.
  • CLUSTER_LIST: if the local CLUSTER_ID is found in the list, the route is discarded.
    *This is done ONLY in RRs.
    This is ONLY created or updated on a RR during Reflection.
Hierarchical Route Reflection:
2 levels:
- level1 RRs are clients of level2 RR -> level1 RR dont need full mesh between them
- level2 RRs are full mesh between them.

Consider:
size of top-level mesh
number of alternative paths

Hierarchical Route Reflection:
2 levels:

– level1 RRs are clients of level2 RR -> level1 RR dont need full mesh between them
– level2 RRs are full mesh between them.

Consider:

  • size of top-level mesh
  • number of alternative paths

RR Design Priciples:

  • keep logical and physical topologies congruent to increase redundancy and path optimization and prevent loops
    — follow physical topology
    — change physical topology
    — modify logical topology
    — follow physical topology
    — session between RR and non-client shouldnt traverse a client
    — session between RR and client shouldnt traverse a non-client
  • use comparable metrics in route selection to avoid convergence oscillations
    (ie MED – only used between prefixes from same neighbor AS – not used for different AS)
    — full ibgp mess –> no
    — always-compare-med –> no
    — deterministic-med -> ok
    — med=0 (via RM in) -> ok
    — bgp communities ->
  • set proper intra and inter cluster IGP metrics to avoid convergence oscillations
    — multicluster RR architecture: intracluster metrics lower than intercluster.
  • modify next-hop with care. Do so only to bring the RRs into forwarding path.
  • use peer groups with RR to reduce convergence time.

GCP Networking 101 – IP Forwarding

I had my shiny and tiny GCP network for EVE-NG to test vEOS. I built a new VM (vm2) to be my center for automation so I can test stuff like ansible/napalm/nornir etc… But I couldn’t ping from vm2 to the vEOS instances in eve-ng (vm1). Those instances where in a different network attached to vm1 so it had to “route”.

As usual, I missed one step when I created the EVE-NG VM. The official documentation doesnt mention anything regarding enabling routing in the VM. As I am not used to Cloud environments, I assume that any simple Linux VM can forward traffic if configured.

Surprise Surprise. In GCP (not sure in other cloud providers), you need to enable “forwarding” during the VM creation and you can’t change that afterwards in any way.

After checking the second guide I followed, I realised that guide mentioned the point to enable forwarding to avoid the same problem I was facing…

So I had to gave up and had to build both VMs from scratch….

But at the end, I have routing enabled in both VMs and I can ping to the vEOS images.

And another annoying thing. I couldnt update the next hop in a static route defined in the VPC. So I had to delete it and create again pointing to the new VM with the vEOS.

And dealing with the internal IPs…

Moving on, quite frustrating day. But learned several things about GCP netwoking.