Very handy:
https://www.digitalocean.com/community/cheatsheets/how-to-use-ansible-cheat-sheet-guide
Author: flipaoXIX
Monitoring Syslog: InfluxDB-Telegraf-Grafana via Ansible role
This a continuation of the last blog entry. This time we are going to gather syslog messages from the monitoring containers and it is going to be deployed by ansible!
As usual, all this is based on Anton’s Karneliuk blog post. All credits to him.
So initially we built a monitoring stack with InfluxDB, Telegraf and Grafana manually to gather and visualise SNMP infor form the Arista cEOS switches.
This time, we are going to send SYSLOG from the monitoring stack containers to a new Telegraf instance.
Ideally, we would like to send Syslog from the cEOS devices but as Anton mentions, the syslog rfc3164 that most network kit implements, it is not supported (yet) by telegraf, that supports rfc5424.
You can read more info about this in all these links:
https://github.com/influxdata/telegraf/issues/4593
https://github.com/influxdata/go-syslog/pull/27
https://github.com/influxdata/telegraf/issues/7023
https://github.com/influxdata/telegraf/issues/4687
https://medium.com/@leodido/from-logs-to-metrics-f38854e3441a
https://itnext.io/metrics-from-kubernetes-logs-82cb1dcb3551
So the new ansible role for building influx-telegraf-grafana instances is “monitoring_stack”:
├── ansible.cfg
├── ansible-hosts
├── group_vars
│ ├── ceoslab.yaml
│ └── monitoring.yaml
└── playbooks
├── monitoring.yaml
└── roles
├── monitoring_stack
│ ├── tasks
│ │ ├── container_grafana.yml
│ │ ├── container_influxdb.yml
│ │ ├── container_telegraf_snmp.yml
│ │ ├── container_telegraf_syslog.yml
│ │ └── main.yml
│ └── templates
│ ├── telegraf_snmp_template.j2
│ └── telegraf_syslog_template.j2
We will have four monitoring containers:
- influxdb: our time-series database with two databases: snmp and syslog
- grafana: GUI to visualize influxdb contents, we will have pales for snmp and syslog queries. It will need to connect to influxdb
- telegraf-snmp: collector of snmp info from the cEOS containers. The list is introduced manually in the template. It will write in influxdb
- telegraf-syslog: collector of syslog messages from the monitoring containers. It will write in influxdb
As the containers are running locally, we define them in the inventory like this:
$ cat ansible-hosts .... [monitoring] localhost
We define some variables too in group_vars for the monitoring containers that will be used in the jinja2 templates and tasks
$ cat group_vars/monitoring.yaml
# Defaults for Docker containers
docker_mon_net:
name: monitoring
subnet: 172.18.0.0/16
gateway: 172.18.0.1
path_to_containers: /PICK_YOUR_PATH/monitoring-example
var_influxdb:
username: xxx
password: xxx123
snmp_community: xxx123
db_name:
snmp: snmp
syslog: syslog
var_grafana:
username: admin
password: xxx123
var_telegraf:
…
So we execute the playbook like this:
ansible master$ ansible-playbook playbooks/monitoring.yaml -vvv --ask-become-pass
The very first time, if you pay attention to the ansible logging, everything should success. If for any reason you have to make changes or troubleshoot, and execute again the full playbook, some tasks will fail, but not the playbook (this is done with ignore_errors: yes inside a task). For example, the docker network creation will fail as it is already there. The same if you try to create the user and dbs in a already running influx instance.
That playbook just calls the role “monitoring_stack“. The main playbook in that role will create the docker network where all containers will be attached, all the containers and do something hacky with iptables.
As the cEOS lab is built (using docker-topo) independently of this playbook, there are already some iptables rules in place, and somehow, when executing the role, the rules change and it blocks the new network for any outbound connectivity.
Before the iptables change in the playbook:
# iptables -t filter -S DOCKER-ISOLATION-STAGE-1 Warning: iptables-legacy tables present, use iptables-legacy to see them -N DOCKER-ISOLATION-STAGE-1 -A DOCKER-ISOLATION-STAGE-1 -i br-4bd17cfa19a8 ! -o br-4bd17cfa19a8 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i br-94c1e813ad6f ! -o br-94c1e813ad6f -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -i br-13ab2b6a0d1d ! -o br-13ab2b6a0d1d -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -i br-00db5844bbb0 ! -o br-00db5844bbb0 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -i br-121978ca0282 ! -o br-121978ca0282 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -j RETURN # # iptables -t filter -S DOCKER-ISOLATION-STAGE-2 Warning: iptables-legacy tables present, use iptables-legacy to see them -N DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-2 -o br-4bd17cfa19a8 -j DROP -A DOCKER-ISOLATION-STAGE-2 -o br-94c1e813ad6f -j DROP -A DOCKER-ISOLATION-STAGE-2 -o br-13ab2b6a0d1d -j DROP -A DOCKER-ISOLATION-STAGE-2 -o br-00db5844bbb0 -j DROP -A DOCKER-ISOLATION-STAGE-2 -o br-121978ca0282 -j DROP -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP -A DOCKER-ISOLATION-STAGE-2 -j RETURN
I want to avoid DOCKER-ISOLATION-STAGE-2 so I want the “-A DOCKER-ISOLATION-STAGE-1 -j ACCEPT” on top of that chain.
This is not the first (neither last) time that this issue bites me. I need to review carefully the docker-topo file and really get me head around the networking expectations from docker.
Another thing about docker networking that bites me very often. In my head, each monitoring has an IP. For example influxdb is 172.18.0.2 and telegraf-syslog is 172.18.0.4. We have configured influxdb to send syslog to telegraf-syslog container so I would expect the influxdb container to use its 0.2 and everything is local (no forwarding, no firewall, etc0. But not, it uses the host ip, 172.18.0.1.
Apart from that, there are several things that I had to review while adapting the role to my environment regarding docker and ansible.
docker documentation:
how to create network: https://docs.docker.com/engine/reference/commandline/network_create/
how to configure container logs: https://docs.docker.com/engine/reference/commandline/container_logs/
how to configure the logging driver in a container: https://docs.docker.com/config/containers/logging/configure/
how to configure syslog in a container: https://docs.docker.com/config/containers/logging/syslog/
how to run commands from a running container: https://docs.docker.com/engine/reference/commandline/exec/
ansible documentation:
become – run comamnds with sudo in a playbook: https://docs.ansible.com/ansible/latest/user_guide/become.html (–ask-become-pass, -K)
docker container module: https://docs.ansible.com/ansible/latest/modules/docker_container_module.html
grafana data source module: https://docs.ansible.com/ansible/latest/modules/grafana_datasource_module.html
This is important because via ansible, I had to workout the meaning of become, how to add the syslog config in the containers and add grafana datasources via a module.
All my ansible code is here.
Another thing I had to hardcode in the code, it is the IP for the telegraf-syslog container in each container playbook:
syslog-address: “udp://172.18.0.4:6514”
$ cat container_influxdb.yml
---
...
- name: 4- CONTAINER INFLUXDB // LAUNCHING CONTAINER
docker_container:
name: influxdb
image: influxdb
state: started
command: "-config /etc/influxdb/influxdb.conf"
networks:
- name: "{{ docker_mon_net.name }}"
purge_networks: yes
ports:
- "8086:8086"
volumes:
- "{{ path_to_containers }}/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf:ro"
- "{{ path_to_containers }}/influxdb/data:/var/lib/influxdb"
log_driver: syslog
log_options:
syslog-address: "udp://172.18.0.4:6514"
tag: influxdb
syslog-format: rfc5424
become: yes
tags:
- tag_influx
...Once you have all containers running:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dd519ff01d6e telegraf "/entrypoint.sh -con…" 4 hours ago Up 4 hours 8092/udp, 0.0.0.0:161->161/udp, 8125/udp, 8094/tcp telegraf_snmp
869f158046a6 grafana/grafana "/run.sh" 4 hours ago Up 4 hours 0.0.0.0:3000->3000/tcp grafana
dc68f261746b influxdb "/entrypoint.sh -con…" 4 hours ago Up 4 hours 0.0.0.0:8086->8086/tcp influxdb
3662c3c69b21 telegraf "/entrypoint.sh -con…" 6 hours ago Up 6 hours 8092/udp, 0.0.0.0:6514->6514/udp, 8125/udp, 8094/tcp telegraf_syslog
ada1f884f1b7 ceos-lab:4.23.3M "/sbin/init systemd.…" 28 hours ago Up 4 hours 0.0.0.0:2002->22/tcp, 0.0.0.0:9002->443/tcp 3node_r03
22d9c4ae9043 ceos-lab:4.23.3M "/sbin/init systemd.…" 28 hours ago Up 4 hours 0.0.0.0:2001->22/tcp, 0.0.0.0:9001->443/tcp 3node_r02
fe7046b1f425 ceos-lab:4.23.3M "/sbin/init systemd.…" 28 hours ago Up 4 hours 0.0.0.0:2000->22/tcp, 0.0.0.0:9000->443/tcp 3node_r01
You should verify that syslog messages are stored in influxdb:
$ curl -G 'https://localhost:8086/query?db=syslog&pretty=true&u=xxx&p=xxx123' --data-urlencode "q=SELECT * FROM syslog limit 2" --insecure
{
"results": [
{
"statement_id": 0,
"series": [
{
"name": "syslog",
"columns": [
"time",
"appname",
"facility",
"facility_code",
"host",
"hostname",
"message",
"msgid",
"procid",
"severity",
"severity_code",
"timestamp",
"version"
],
"values": [
[
"2020-07-21T12:08:16.216632823Z",
"influxdb",
"daemon",
3,
"3662c3c69b21",
"athens",
"ts=2020-07-21T12:08:16.169711Z lvl=info msg=\"InfluxDB starting\" log_id=0O8KE_AG000 version=1.8.1 branch=1.8 commit=af0237819ab9c5997c1c0144862dc762b9d8fc25",
"influxdb",
"11254",
"err",
3,
1595333296000000000,
1
],
We can create the new queries in grafana for SYSLOG. The datasources are already created by ansible so we dont have to worry about that.
For creating a query about the number of syslog messages we receive. This is what I did:
Most of the entries come from “influxdb”.
For creating a query with the content of each syslog message:
Here I struggled a bit. I can’t really change much in the table view.
And this is the dashboard with the syslog queries and snmp from the last blog entry:
So at the end, I have an ansible role working!
Need to learn more about how to backup stuff from grafana. I have been playing with this:
https://github.com/ysde/grafana-backup-tool
Next thing I want to try is telemetry.
Monitoring: InfluxDB-Telegraf-Grafana
This is something I wanted to try for some time. Normally for networks monitoring you use a NMS tool. They can be expensive, free or cheap. I have seen/used Observium and LibreNMS. And many years ago Cacti. There are other tools that can do the job like Zabbix/Nagios/Icinga.
But it seems time-series-databases are the new standard. They give you more flexibility as you can create queries and graph them.
There are many tools out there that I dont really know like Prometheus, the elk-stack (Elasticsearch, Logstash, and Kibana), Influxdb, telegraf and grafana.
I decided for InfluxDB-Telegraf-Grafana stuck as I could find quickly info based on scenarios of networks.
What is the rule of eachc one:
Telegraf: collect data
InfluxDB: store data
Grafana: visualize
My main source is again Anton’s blog. All credits to him.
Environment
My network is just 3 Arista ceos containers via docker. All services will run as containers so you need docker installed. Everything is IPv4.
InfluxDB
Installation:
// Create directories
mkdir telemetry-example/influxdb
cd telemetry-example/influxdb
// Get influxdb config
docker run --rm influxdb influxd config > influxdb.conf
// Create local data folder for influxdb that we will map
mkdir data
ls -ltr
// Check docker status
docker images
docker ps -a
// Create docker instance for influxdb. Keep in mind that I am giving a name to the instance
docker run -d -p 8086:8086 -p 8088:8088 --name influxdb \
-v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf:ro \
-v $PWD/data:/var/lib/influxdb \
influxdb -config /etc/influxdb/influxdb.conf
// Verify connectivity
curl -i http://localhost:8086/ping
// Create database "test" using http-query (link below for more details)
curl -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE test"
{"results":[{"statement_id":0}]} <-- command was ok!
// Create user/pass for your db.
curl -XPOST http://localhost:8086/query --data-urlencode "q=CREATE USER xxx WITH PASSWORD 'xxx123' WITH ALL PRIVILEGES"
{"results":[{"statement_id":0}]} <-- command was ok!
// Create SSL cert for influxdb
docker exec -it influxdb openssl req -x509 -nodes -newkey rsa:2048 -keyout /etc/ssl/influxdb-selfsigned.key -out /etc/ssl/influxdb-selfsigned.crt -days 365 -subj "/C=GB/ST=LDN/L=LDN/O=domain.com/CN=influxdb.domain.com"
// Update influxdb.conf for SSL
telemetry-example/influxdb$ vim influxdb.conf
…
https-enabled = true
https-certificate = "/etc/ssl/influxdb-selfsigned.crt"
https-private-key = "/etc/ssl/influxdb-selfsigned.key"
…
// Restart influxdb to take the changes
docker restart influxdb
// Get influxdb IP for using it later
docker container inspect influxdb --format='{{ .NetworkSettings.IPAddress }}'
172.17.0.2
// Verify connectivity via https
curl -i https://localhost:8086/ping --insecure
The verification for HTTPS was a bit more difficult because the result was always correct no matter what query I was running:
$ curl -G https://localhost:8086/query --data-urlencode "db=test" --data-urlencode "q=SELECT * FROM test" --insecure
{"results":[{"statement_id":0}]}
$ curl -XPOST 'https://localhost:8086/query?db=test&u=xxx&p=xxx123' --data-urlencode 'q=SELECT * FROM test' --insecure
{"results":[{"statement_id":0}]}
$ curl -XPOST 'https://localhost:8086/query?db=test&u=xxx&p=yyy1231' --data-urlencode 'q=SELECT * FROM test' --insecure
{"results":[{"statement_id":0}]}
So I decided to see if there was cli/shell for the influxdb (like in mysql, etc). And yes, there is one. Keep in mind that you have to use “-ssl -unsafeSsl” at the same time! That confused me a lot.
$ docker exec -it influxdb influx -ssl -unsafeSsl Connected to https://localhost:8086 version 1.8.1 InfluxDB shell version: 1.8.1 > show databases name: databases name _internal test > use test Using database test > show series key cpu,cpu=cpu-total,host=5f7aa2c5550e
Links about influxdb that are good for the docker creation and the http queries:
https://hub.docker.com/_/influxdb
https://docs.influxdata.com/influxdb/v1.7/tools/api/#query-http-endpoin
Telegraf
I struggled with the SNMP config needed in Telegraf. The installation was fine.
Links:
https://hub.docker.com/_/telegraf
https://docs.influxdata.com/telegraf/v1.14/introduction/getting-started/
Steps:
// Create dir mkdir telemetry-example/telegraf cd telemetry-example/telegraf // Get config file to be modified docker run --rm telegraf telegraf config > telegraf.conf // Add the details of influxdb in telegraf.conf. As well, you need to add the devices you want to poll. In my case 172.23.0.2/3/4. vim telegraf.conf .... [[outputs.influxdb]] urls = ["https://172.17.0.2:8086"] database = "test" skip_database_creation = false ## Timeout for HTTP messages. timeout = "5s" ## HTTP Basic Auth username = "xxx" password = "xxx123" ## Use TLS but skip chain & host verification insecure_skip_verify = true # Retrieves SNMP values from remote agents [[inputs.snmp]] ## Agent addresses to retrieve values from. ## example: agents = ["udp://127.0.0.1:161"] ## agents = ["tcp://127.0.0.1:161"]agents = ["udp://172.23.0.2:161","udp://172.23.0.3:161","udp://172.23.0.4:161"]# ## Timeout for each request.timeout = "5s"# ## SNMP version; can be 1, 2, or 3.version = 2# ## SNMP community string.community = "tomas123"# ## Number of retries to attempt.retries = 3
This is the SNMP config I added below the SNMPv3 options in [[inputs.snmp]]
# ## Add fields and tables defining the variables you wish to collect. This
# ## example collects the system uptime and interface variables. Reference the
# ## full plugin documentation for configuration details.
[[inputs.snmp.field]]
name = "hostname"
oid = "RFC1213-MIB::sysName.0"
is_tag = true
[[inputs.snmp.field]]
name = "uptime"
oid = "DISMAN-EVENT-MIB::sysUpTimeInstance"
# IF-MIB::ifTable contains counters on input and output traffic as well as errors and discards.
[[inputs.snmp.table]]
name = "interface"
inherit_tags = [ "hostname" ]
oid = "IF-MIB::ifTable"
# Interface tag - used to identify interface in metrics database
[[inputs.snmp.table.field]]
name = "ifDescr"
oid = "IF-MIB::ifDescr"
is_tag = true
# IF-MIB::ifXTable contains newer High Capacity (HC) counters that do not overflow as fast for a few of the ifTable counters
[[inputs.snmp.table]]
name = "interfaceX"
inherit_tags = [ "hostname" ]
oid = "IF-MIB::ifXTable"
# Interface tag - used to identify interface in metrics database
[[inputs.snmp.table.field]]
name = "ifDescr"
oid = "IF-MIB::ifDescr"
is_tag = true
# EtherLike-MIB::dot3StatsTable contains detailed ethernet-level information about what kind of errors have been logged on an interface (such as FCS error, frame too long, etc)
[[inputs.snmp.table]]
name = "interface"
inherit_tags = [ "hostname" ]
oid = "EtherLike-MIB::dot3StatsTable"
# Interface tag - used to identify interface in metrics database
[[inputs.snmp.table.field]]
name = "name"
oid = "IF-MIB::ifDescr"
is_tag = true
For more info about the SNMP config in telegraf. These are good links. This is the official github page. And this is the page for SNMP input plugin that explain the differences between “field” and “table”.
As well, the link below is really good too for explaining the SNMP config in telegraf:”Gathering Data via SNMP”
https://blog.networktocode.com/post/network_telemetry_for_snmp_devices/
Start the container:
docker run -d -p 8125:8125 -p 8092:8092 -p 8094:8094 --name telegraf \ -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro \ telegraf -config /etc/telegraf/telegraf.conf
Check the logs:
docker logs telegraf -f ... 2020-07-17T12:45:10Z E! [inputs.snmp] Error in plugin: initializing table interface: translating: exit status 2: MIB search path: /root/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp Cannot find module (EtherLike-MIB): At line 0 in (none) EtherLike-MIB::dot3StatsTable: Unknown Object Identifier ...
You will see errors about not able to find the MIB files! So I used Librenms mibs. I download the project and copied the MIBS I thought I needed (arista and some other that dont belong to a vendor). As well, this is noted by Anton’s in this link:
https://github.com/librenms/librenms/tree/master/mibs
In my case:
/usr/share/snmp/mibs$ ls -ltr
total 4672
-rw-r--r-- 1 root root 52820 Feb 7 2019 UCD-SNMP-MIB.txt
-rw-r--r-- 1 root root 18274 Feb 7 2019 UCD-SNMP-MIB-OLD.txt
-rw-r--r-- 1 root root 8118 Feb 7 2019 UCD-IPFWACC-MIB.txt
-rw-r--r-- 1 root root 6476 Feb 7 2019 UCD-IPFILTER-MIB.txt
-rw-r--r-- 1 root root 3087 Feb 7 2019 UCD-DLMOD-MIB.txt
-rw-r--r-- 1 root root 4965 Feb 7 2019 UCD-DISKIO-MIB.txt
-rw-r--r-- 1 root root 2163 Feb 7 2019 UCD-DEMO-MIB.txt
-rw-r--r-- 1 root root 5039 Feb 7 2019 NET-SNMP-VACM-MIB.txt
-rw-r--r-- 1 root root 4814 Feb 7 2019 NET-SNMP-TC.txt
-rw-r--r-- 1 root root 1226 Feb 7 2019 NET-SNMP-SYSTEM-MIB.txt
-rw-r--r-- 1 root root 2504 Feb 7 2019 NET-SNMP-PERIODIC-NOTIFY-MIB.txt
-rw-r--r-- 1 root root 3730 Feb 7 2019 NET-SNMP-PASS-MIB.txt
-rw-r--r-- 1 root root 1215 Feb 7 2019 NET-SNMP-MONITOR-MIB.txt
-rw-r--r-- 1 root root 2036 Feb 7 2019 NET-SNMP-MIB.txt
-rw-r--r-- 1 root root 9326 Feb 7 2019 NET-SNMP-EXTEND-MIB.txt
-rw-r--r-- 1 root root 9160 Feb 7 2019 NET-SNMP-EXAMPLES-MIB.txt
-rw-r--r-- 1 root root 15901 Feb 7 2019 NET-SNMP-AGENT-MIB.txt
-rw-r--r-- 1 root root 5931 Feb 7 2019 LM-SENSORS-MIB.txt
-rw-r--r-- 1 root root 1913 Jul 2 04:38 GNOME-SMI.txt
-rw-r--r-- 1 root root 5775 Jul 17 13:14 SNMPv2-TM
-rw-r--r-- 1 root root 2501 Jul 17 13:14 SNMPv2-TC-v1
-rw-r--r-- 1 root root 38034 Jul 17 13:14 SNMPv2-TC
-rw-r--r-- 1 root root 1371 Jul 17 13:14 SNMPv2-SMI-v1
-rw-r--r-- 1 root root 8924 Jul 17 13:14 SNMPv2-SMI
-rw-r--r-- 1 root root 29305 Jul 17 13:14 SNMPv2-MIB
-rw-r--r-- 1 root root 8263 Jul 17 13:14 SNMPv2-CONF
-rw-r--r-- 1 root root 17177 Jul 17 13:14 INET-ADDRESS-MIB
-rw-r--r-- 1 root root 71691 Jul 17 13:14 IF-MIB
-rw-r--r-- 1 root root 3129 Jul 17 13:15 ARISTA-BGP4V2-TC-MIB
-rw-r--r-- 1 root root 64691 Jul 17 13:15 ARISTA-BGP4V2-MIB
-rw-r--r-- 1 root root 7155 Jul 17 13:15 ARISTA-VRF-MIB
-rw-r--r-- 1 root root 1964 Jul 17 13:15 ARISTA-SMI-MIB
-rw-r--r-- 1 root root 10901 Jul 17 13:15 ARISTA-NEXTHOP-GROUP-MIB
-rw-r--r-- 1 root root 5826 Jul 17 13:15 ARISTA-IF-MIB
-rw-r--r-- 1 root root 4547 Jul 17 13:15 ARISTA-GENERAL-MIB
-rw-r--r-- 1 root root 7014 Jul 17 13:15 ARISTA-ENTITY-SENSOR-MIB
-rw-r--r-- 1 root root 62277 Jul 17 13:21 IANA-PRINTER-MIB
-rw-r--r-- 1 root root 36816 Jul 17 13:21 IANA-MAU-MIB
-rw-r--r-- 1 root root 4299 Jul 17 13:21 IANA-LANGUAGE-MIB
-rw-r--r-- 1 root root 13954 Jul 17 13:21 IANA-ITU-ALARM-TC-MIB
-rw-r--r-- 1 root root 35628 Jul 17 13:21 IANAifType-MIB
-rw-r--r-- 1 root root 15150 Jul 17 13:21 IANA-GMPLS-TC-MIB
-rw-r--r-- 1 root root 10568 Jul 17 13:21 IANA-CHARSET-MIB
-rw-r--r-- 1 root root 4743 Jul 17 13:21 IANA-ADDRESS-FAMILY-NUMBERS-MIB
-rw-r--r-- 1 root root 3518 Jul 17 13:21 IANA-RTPROTO-MIB
-rw-r--r-- 1 root root 13100 Jul 17 13:31 ENTITY-STATE-MIB
-rw-r--r-- 1 root root 16248 Jul 17 13:31 ENTITY-SENSOR-MIB
-rw-r--r-- 1 root root 59499 Jul 17 13:31 ENTITY-MIB
-rw-r--r-- 1 root root 2114 Jul 17 13:31 BGP4V2-TC-MIB
-rw-r--r-- 1 root root 50513 Jul 17 13:31 BGP4-MIB
-rw-r--r-- 1 root root 96970 Jul 17 13:31 IEEE8021-CFMD8-MIB
-rw-r--r-- 1 root root 86455 Jul 17 13:31 IEEE8021-BRIDGE-MIB
-rw-r--r-- 1 root root 113507 Jul 17 13:31 IEEE802171-CFM-MIB
-rw-r--r-- 1 root root 6321 Jul 17 13:31 ENTITY-STATE-TC-MIB
-rw-r--r-- 1 root root 112096 Jul 17 13:31 IEEE802dot11-MIB
-rw-r--r-- 1 root root 44559 Jul 17 13:31 IEEE8023-LAG-MIB
-rw-r--r-- 1 root root 24536 Jul 17 13:31 IEEE8021-TC-MIB
-rw-r--r-- 1 root root 62182 Jul 17 13:31 IEEE8021-SECY-MIB
-rw-r--r-- 1 root root 96020 Jul 17 13:31 IEEE8021-Q-BRIDGE-MIB
-rw-r--r-- 1 root root 62591 Jul 17 13:31 IEEE8021-PAE-MIB
-rw-r--r-- 1 root root 135156 Jul 17 13:31 IEEE8021-CFM-MIB
-rw-r--r-- 1 root root 48703 Jul 17 13:31 IPV6-MIB
-rw-r--r-- 1 root root 15936 Jul 17 13:31 IPV6-ICMP-MIB
-rw-r--r-- 1 root root 3768 Jul 17 13:31 IPV6-FLOW-LABEL-MIB
-rw-r--r-- 1 root root 31323 Jul 17 13:31 IPMROUTE-STD-MIB
-rw-r--r-- 1 root root 33626 Jul 17 13:31 IPMROUTE-MIB
-rw-r--r-- 1 root root 186550 Jul 17 13:31 IP-MIB
-rw-r--r-- 1 root root 46366 Jul 17 13:31 IP-FORWARD-MIB
-rw-r--r-- 1 root root 240526 Jul 17 13:31 IEEE-802DOT17-RPR-MIB
-rw-r--r-- 1 root root 4400 Jul 17 13:31 IPV6-UDP-MIB
-rw-r--r-- 1 root root 7257 Jul 17 13:31 IPV6-TCP-MIB
-rw-r--r-- 1 root root 2367 Jul 17 13:31 IPV6-TC
-rw-r--r-- 1 root root 14758 Jul 17 13:31 IPV6-MLD-MIB
-rw-r--r-- 1 root root 69938 Jul 17 13:31 MPLS-LSR-MIB
-rw-r--r-- 1 root root 61017 Jul 17 13:31 MPLS-L3VPN-STD-MIB
-rw-r--r-- 1 root root 16414 Jul 17 13:31 LLDP-V2-TC-MIB.mib
-rw-r--r-- 1 root root 77651 Jul 17 13:31 LLDP-V2-MIB.mib
-rw-r--r-- 1 root root 76945 Jul 17 13:31 LLDP-MIB
-rw-r--r-- 1 root root 59110 Jul 17 13:31 LLDP-EXT-MED-MIB
-rw-r--r-- 1 root root 30192 Jul 17 13:31 LLDP-EXT-DOT3-MIB
-rw-r--r-- 1 root root 30182 Jul 17 13:31 LLDP-EXT-DOT1-MIB
-rw-r--r-- 1 root root 36147 Jul 17 13:31 LLDP-EXT-DCBX-MIB
-rw-r--r-- 1 root root 145796 Jul 17 13:31 ISIS-MIB
-rw-r--r-- 1 root root 28564 Jul 17 13:31 TCP-MIB
-rw-r--r-- 1 root root 1291 Jul 17 13:31 RFC-1215
-rw-r--r-- 1 root root 79667 Jul 17 13:31 RFC1213-MIB
-rw-r--r-- 1 root root 2866 Jul 17 13:31 RFC-1212
-rw-r--r-- 1 root root 3067 Jul 17 13:31 RFC1155-SMI
-rw-r--r-- 1 root root 60053 Jul 17 13:31 MPLS-VPN-MIB
-rw-r--r-- 1 root root 95418 Jul 17 13:31 MPLS-TE-STD-MIB
-rw-r--r-- 1 root root 55490 Jul 17 13:31 MPLS-TE-MIB
-rw-r--r-- 1 root root 26327 Jul 17 13:31 MPLS-TC-STD-MIB
-rw-r--r-- 1 root root 76361 Jul 17 13:31 MPLS-LSR-STD-MIB
-rw-r--r-- 1 root root 12931 Jul 17 13:31 RFC1389-MIB
-rw-r--r-- 1 root root 30091 Jul 17 13:31 RFC1284-MIB
-rw-r--r-- 1 root root 147614 Jul 17 13:31 RFC1271-MIB
-rw-r--r-- 1 root root 22342 Jul 17 13:33 SNMP-FRAMEWORK-MIB
-rw-r--r-- 1 root root 223833 Jul 17 13:33 RMON2-MIB
-rw-r--r-- 1 root root 127407 Jul 17 13:33 DIFFSERV-MIB
-rw-r--r-- 1 root root 101324 Jul 17 13:34 TOKEN-RING-RMON-MIB
-rw-r--r-- 1 root root 147822 Jul 17 13:34 RMON-MIB
-rw-r--r-- 1 root root 26750 Jul 17 13:34 INTEGRATED-SERVICES-MIB
-rw-r--r-- 1 root root 1863 Jul 17 13:34 DIFFSERV-DSCP-TC
-rw-r--r-- 1 root root 34162 Jul 17 13:34 SNMP-VIEW-BASED-ACM-MIB
-rw-r--r-- 1 root root 84133 Jul 17 13:34 Q-BRIDGE-MIB
-rw-r--r-- 1 root root 16414 Jul 17 13:34 TRANSPORT-ADDRESS-MIB
-rw-r--r-- 1 root root 39879 Jul 17 13:35 P-BRIDGE-MIB
-rw-r--r-- 1 root root 4660 Jul 17 13:35 HCNUM-TC
-rw-r--r-- 1 root root 54884 Jul 17 13:35 BRIDGE-MIB
-rw-r--r-- 1 root root 2628 Jul 17 13:35 VPN-TC-STD-MIB
-rw-r--r-- 1 root root 10575 Jul 17 13:35 HC-PerfHist-TC-MIB
-rw-r--r-- 1 root root 22769 Jul 17 13:50 SNMP-TARGET-MIB
-rw-r--r-- 1 root root 84492 Jul 17 13:54 EtherLike-MIB
-rw-r--r-- 1 root root 68177 Jul 17 13:56 DISMAN-EXPRESSION-MIBOnce you have the MIB files, you need to copy across to the telegraf container:
docker cp /usr/share/snmp/mibs/. telegraf:/usr/share/snmp/mibs/ docker restart telegraf
These two links helped me a lot to troubleshoot and understand snmpwalk:
https://www.dev-eth0.de/2016/12/06/grafana_snmp/
If you dont want to see the OIDS in snmpwalk, you need to load all your MIBS with -m ALL
/usr/share/snmp/mibs$ snmpwalk -Os -v2c -c community -m ALL 172.23.0.2 | head -10 sysDescr.0 = STRING: Linux r01 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64 sysObjectID.0 = OID: aristaProducts.2600 sysUpTimeInstance = Timeticks: (2503676) 6:57:16.76 sysContact.0 = STRING: sysName.0 = STRING: r01 sysLocation.0 = STRING: ceoslab sysServices.0 = INTEGER: 14 sysORLastChange.0 = Timeticks: (35427) 0:05:54.27 sysORID.1 = OID: tcpMIB sysORID.2 = OID: mib-2.50 $ snmpwalk -v2c -c community -m ALL 172.23.0.2 | head -10 SNMPv2-MIB::sysDescr.0 = STRING: Linux r01 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64 SNMPv2-MIB::sysObjectID.0 = OID: ARISTA-SMI-MIB::aristaProducts.2600 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (2535736) 7:02:37.36 SNMPv2-MIB::sysContact.0 = STRING: SNMPv2-MIB::sysName.0 = STRING: r01 SNMPv2-MIB::sysLocation.0 = STRING: ceoslab SNMPv2-MIB::sysServices.0 = INTEGER: 14 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (35427) 0:05:54.27 SNMPv2-MIB::sysORID.1 = OID: TCP-MIB::tcpMIB SNMPv2-MIB::sysORID.2 = OID: SNMPv2-SMI::mib-2.50 /usr/share/snmp/mibs$ snmpwalk -Os -v2c -c community 172.23.0.2 | head -10 iso.3.6.1.2.1.1.1.0 = STRING: "Linux r01 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.30065.1.2600 iso.3.6.1.2.1.1.3.0 = Timeticks: (2505381) 6:57:33.81 iso.3.6.1.2.1.1.4.0 = "" iso.3.6.1.2.1.1.5.0 = STRING: "r01" iso.3.6.1.2.1.1.6.0 = STRING: "ceoslab" iso.3.6.1.2.1.1.7.0 = INTEGER: 14 iso.3.6.1.2.1.1.8.0 = Timeticks: (35427) 0:05:54.27 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.2.1.49 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.2.1.50
And if you want to verify that telegraf is capable to use the MIBS:
docker exec -it telegraf snmpwalk -Os -v2c -c community -m ALL 172.23.0.2 | head -10
Now, you can check if telegraf is updating influxdb. If there is output, it is good!
$ curl -G 'https://localhost:8086/query?db=test&pretty=true&u=xxx&p=xxx123' --data-urlencode "q=SELECT * FROM interfaceX limit 2" --insecure
{
"results": [
{
"statement_id": 0,
"series": [
{
"name": "interfaceX",
"columns": [
"time",
"agent_host",
"host",
"hostname",
"ifAlias",
"ifConnectorPresent",
"ifCounterDiscontinuityTime",
"ifDescr",
"ifHCInBroadcastPkts",
"ifHCInMulticastPkts",
"ifHCInOctets",
"ifHCInUcastPkts",
"ifHCOutBroadcastPkts",
"ifHCOutMulticastPkts",
"ifHCOutOctets",
"ifHCOutUcastPkts",
"ifHighSpeed",
"ifInBroadcastPkts",
"ifInMulticastPkts",
"ifLinkUpDownTrapEnable",
"ifName",
"ifOutBroadcastPkts",
"ifOutMulticastPkts",
"ifPromiscuousMode",
"name"
],
"values": [
[
"2020-07-17T13:00:10Z",
"172.23.0.2",
"6778bdf4ea85",
"r01",
null,
1,
0,
"Ethernet1",
0,
2118,
2013125,
7824,
0,
0,
0,
0,
0,
0,
2118,
1,
"Ethernet1",
0,
0,
2,
null
],
[
"2020-07-17T13:00:10Z",
"172.23.0.2",
"6778bdf4ea85",
"r01",
"CORE Loopback",
2,
0,
"Loopback1",
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
"Loopback1",
0,
0,
2,
null
]
]
}
]
}
]
}
Grafana
I have seen Grafana before but I have never used it so the configuration on queries was a bit of a challenge but I was lucky and I found very good blogs for that. The installation process is ok:
// Create folder for grafana and data mkdir -p telemetry-example/grafana/data cd telemetry-example/grafana // Create docker instance docker run -d -p 3000:3000 --name grafana \ --user root \ -v $PWD/data:/var/lib/grafana \ grafana/grafana // Create SSL cert for grafana docker exec -it grafana openssl req -x509 -nodes -newkey rsa:2048 -keyout /etc/ssl/grafana-selfsigned.key -out /etc/ssl/grafana-selfsigned.crt -days 365 -subj "/C=GB/ST=LDN/L=LDN/O=domain.com/CN=grafana.domain.com" // Copy grafana config so we can update it docker cp grafana:/etc/grafana/grafana.ini grafana.ini // Update grafana config with SSL vim grafana.ini ############################## Server [server] # Protocol (http, https, h2, socket) protocol = https … # https certs & key file cert_file = /etc/ssl/grafana-selfsigned.crt cert_key = /etc/ssl/grafana-selfsigned.key // Copy back the config to the container and restart docker cp grafana.ini grafana:/etc/grafana/grafana.ini docker container restart grafana
Now you can open in your browser to grafana “https://0.0.0.0:3000/ ” using admin/admin
You need to add a data source that is our influxdb container. So you need to pick up the “influxdb” type and fill the values as per below.
Now, you need to create a dashboard with panel.
Links that I reviewed for creating the dasbord
For creating a panel. The link below was the best on section “Interface Throughput”. Big thanks to the author.
https://lkhill.com/telegraf-influx-grafana-network-stats/
This is my query for checking all details:
And this is my final dashboard.
SNMP Config
BTW, you need to config SNMP in the switches so telegraf can poll it:
snmp-server location ceoslab snmp-server community xxx123 ro snmp-server host 172.17.0.1 version 2c xxx123
In my case, the stack of containers Influx-Telegraf-Grafana are running on the default bridge. Each container has its own IP but as the Arista containers are in the different docker network, it needs to “route” so the IP of telegraf container will be NAT-ed to 172.17.0.1 from the switches point of view.
Next
I would like to manage all this process via Ansible… Something like this.. but will take me time
Notes
As usual, I have struggled but I have learned a lot and at the end things are working. I am happy with that.
Debian – apt-file
In the past, I have had to use Centos systems a lot at work and there was something I really liked from rpm, it is “yum provides” that tells you which package you need to install based on the command you need.
I always struggle to do that in Debian. I hope I remember it for the next time. Based on this link:
# aptitude install apt-file # apt-file update # apt-file search snmpwalk libnet-snmp-perl: /usr/share/doc/libnet-snmp-perl/examples/snmpwalk.pl libsnmp-session-perl: /usr/share/doc/libsnmp-session-perl/examples/snmpwalkh.pl python3-pysnmp4-apps: /usr/bin/pysnmpwalk python3-pysnmp4-apps: /usr/share/man/man1/pysnmpwalk.1.gz snmp: /usr/bin/snmpwalk <=== THIS IS WHAT I WANT !!!! snmp: /usr/share/man/man1/snmpwalk.1.gz snmpsim: /usr/share/doc/snmpsim/examples/data/foreignformats/linux.snmpwalk.gz snmpsim: /usr/share/doc/snmpsim/examples/data/foreignformats/winxp1.snmpwalk.gz # aptitude install snmp
Ansible Troubleshooting 2
Today I was trying to write a playbook to push config to Arista devices.
Initially I wanted to use napalm module to push the config (as I have done with nornir) but it seems the napalm-ansible module requires napalm3 and netmiko3 and that breaks my nornir2.4 ( that requires napalm<3) So I uninstalled napalm-ansible and restored the other packages. Good thing i chekced the version before hand.
$ python -m pip list | grep -E 'nornir|napalm|netmiko|ansible' ansible 2.9.10 napalm 2.5.0 netmiko 2.4.2 nornir 2.4.0
So I had to check the eos_config module. I think the napalm-ansible module is more powerful as it uses diff and sessions provided by Arista. As far as I can see, there is no option to say to the module to just make a dry run.
At the end I managed to put everything together but the eos_config was failing:
TASK [11- push config]
task path: xxx/testdir2/ceos-testing/ansible/playbooks/gen-config.yaml:60
fatal: [r1]: FAILED! => {
"changed": false,
"msg": "path specified in src not found"
}
The funny thing is all other tasks that needed to use templates were using the same path and were fine:
- name: 10- merge all configs in one file
assemble:
src: "CFGS/{{ inventory_hostname }}/"
dest: "CFGS/{{ inventory_hostname }}-full.txt"
- name: 11- push config
debugger: on_failed
eos_config:
#src: "{{playbook_dir}}/../CFGS/{{ inventory_hostname }}-full.txt"
src: "../CFGS/{{ inventory_hostname }}-full.txt"
backup: yesSo I had to find out where that task was looking for the file. It seems “assemble“, “template” and “file” tasks use as pwd where I am calling the script (xxx/testdir2/ceos-testing/ansible). But “eos_config” is using where the playbook is (xxx/testdir2/ceos-testing/ansible/playbook) based on my running command “…/ansible master$ ansible-playbook playbooks/gen-config.yaml“.
So I was searching for some help and I found the playbook path and ansible search paths. So now I needed to verify that. I found some ansible debugger and examples that were really useful!
So I used “debugger: on_failed” for my task 11. And could see the path:
TASK [11- push config]
task path: /home/tomas/storage/technology/arista/testdir2/ceos-testing/ansible/playbooks/gen-config.yaml:60
fatal: [r1]: FAILED! => {
"changed": false,
"msg": "path specified in src not found"
}
[r1] TASK: 11- push config (debug)> p task.args
{'backup': True,
'src': '/home/tomas/storage/technology/arista/testdir2/ceos-testing/ansible/playbooks/CFGS/r1-full.txt'}
[r1] TASK: 11- push config (debug)> quit
User interrupted execution
So it is clear it was looking at the playbook dir.
So after fixing the path, I realised that I didn’t want to run everything and wanted to use tags so only the last part was executed.
/ansible master$ cat playbooks/gen-config.yaml ...- name: 12- display resultdebug: msg: "Backup file is {{ load_config.shortname }} and result is: {{ load_config }}"tags: push_config... /ansible master$ ansible-playbook playbooks/gen-config.yaml --limit="r1" -vvv --tags "push_config"
One more thing, the output of ansible when you have dictionaries, it is not great. I checked this link and it is good for failures and with -vvvv. But for green outputs still not great:
TASK [12- - display result] *
task path: /home/tomas/storage/technology/arista/testdir2/ceos-testing/ansible/playbooks/gen-config.yaml:61
ok: [r1] =>
msg: 'Backup file is /home/tomas/storage/technology/arista/testdir2/ceos-testing/ansible/playbooks/backup/r1_config and result is: {''changed'': True, ''commands'': [''interface Ethernet1'', ''no shutdown'', ''interface Ethernet2'', ''no shutdown'', ''router bgp 100'', ''neighbor AS100-CORE password mpls-sr''], ''updates'': [''interface Ethernet1'', ''no shutdown'', ''interface Ethernet2'', ''no shutdown'', ''router bgp 100'', ''neighbor AS100-CORE password mpls-sr''], ''session'': ''ansible_1594920727'', ''backup_path'': ''/home/tomas/storage/technology/arista/testdir2/ceos-testing/ansible/playbooks/backup/r1_config.2020-07-16@18:32:07'', ''date'': ''2020-07-16'', ''time'': ''18:32:07'', ''shortname'': ''/home/tomas/storage/technology/arista/testdir2/ceos-testing/ansible/playbooks/backup/r1_config'', ''filename'': ''r1_config.2020-07-16@18:32:07'', ''failed'': False}'
META: ran handlers
META: ran handlers
PLAY RECAP
r1 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
(testdir2) go:1.12.5|py:3.7.3|tomas@athens:~/storage/technology/arista/testdir2/ceos-testing/ansible master$
Jinja2
Nice Jinja2 parser online! Great job for the author!!!
From this link:
https://ttl255.com/jinja2-tutorial-part-3-whitespace-control/
Traceroute
A good refresh about traceroute. It is a very common tool for network troubleshooting so it is important to use it wisely
Important points
- ICMP vs UDP: most implementations do UDP (it can be blocked…)
- Every probe is an independent trial!
- Try to identify the characteristics and location of each hop
- If there is a congestion/delay issue in one hop, it has to be carried out to the next hops, if not, it is just prioritizing of the ICMP generation by that router/hop.
- You dont see the reverse path – Ask the other end (if possible) to send the traceroute from its end.
- Border routers between providers can be a hot spot for issues.
- Asymmetric paths can bite you. Try to set the source address in your tests (from the provider IP, from your own space, etc)
- Spot ECMP (in the same hop, you see several different IPs). Multiple unequal length paths can be painful.
- MPLS: most times is hidden (TTL is removed). It can be tricky to spot. But it can be funny when you see the hops (with private IPs 🙂
And if you are more interested in the paths than latency, this can be a good too:
Breath and Cold
I finished this book. It is quite good, I have read about Wim Hof through another book that mentioned the benefits of cold for physical recovery in sports. And after a bit of more searching I was interested in reading more.
The book shows the benefits of breathing and how disconnected we are from our environment. And we have the tools to change that.
In other books about depression and trauma, mentioned the disconnection from our environment as one main cause, and getting back to it (nature) gets improvements.
I have never been a strong swimmer neither have a big lung capacity but I remember that when I was swimming as a teenager in my hometown, after doing the breathing exercises I could hold much longer. And the second day I tried the breathing exercise from the book, I managed 4 minutes. I was quite surprised. So I am adding this to my meditation practice too.
Regarding the cold, the days I am going for a run, I finish my shower with cold water (based on the fittest book) so I can give better recovery to my legs (and knees). And to be honest, It feels very good afterwards. If you don’t fight the cold (don’t shiver) it is interesting how your body relaxes and heartbeat slows down.
So I will carry on with the cold and breath exercises.
VIM – Indent multiple lines quickly
This is the thing I have to do from time to time and I never remember how to do:
https://stackoverflow.com/questions/235839/indent-multiple-lines-quickly-in-vi
And this what works for me:
Key presses for more visual people: 1- Enter Command Mode: Escape 2- Move around to the start of the area to indent: hjkl↑↓←→ 3- Start a block: v 4- Move around to the end of the area to indent: hjkl↑↓←→ 5- Type the number of indentation levels you want 0..9 6- Execute the indentation on the block: >
Let’s see if I can remember it…
Openconfig
I have struggled to get something working for learning a bit of openconfig.
At the end, all info my info comes from Anton’s Karneliuk blog series about Openconfig. So all credit to him. It is the best source about real testing of openconfig in different platforms.
I am not going to create the wheel and explain what openconfig is. In my head, it is attempt from several big vendors to standardise the network management (config) and monitoring (telemetry) via YANG (vendor-neutral) models. So OC uses YANG. And we interact with OC using a transport protocol like netconf, restcong and gNMI. So the network devices need to implement one of these protocols. Based on the blog Cisco, Nokia and Arista have netconf implementations and Ansible has a module for that!!! So the key words are openconfig, yang and netcong.
So in my case, based on my ceos lab, I have added a new playbook based on Anton’s to test openconfig/netcong with Arista cEOS:
https://github.com/thomarite/ceos-testing/blob/master/README.md#openconfig
This is quite basic as it only gets the interface config.
Following Anton’s Part3 Blog:
I tried to push config via openconfig to my ceos devices (all files are in github as per link above).
The blog is dense but it is good because there is a lot of info. In this case, you have to use an Ansible role so it is a new thing to learn. As well, I wanted to adapt that role to my env and found some ansible issues but managed to fix after reading ansible documentation and paying attention to the -vvv info.
From “oc-push-config.yaml” the first task “collect” it is fine. It just takes some 10 minutes or more to get all YANG modules from each device.
The issue is with task “configure”. It fails when trying to push the interface config. I have tried Anton’s config and the actual config generated from oc-get-interface-info.yaml but no joy.
Based on the blog, it seems Arista doesnt have much interest in Openconfig.
Anyway, there have been a couple of intense days looking at all this openconfig/netcong/yang thing. I have just touched the surface but I have learned some Ansible in the way too. So could be worse.
Will move on to something else.