{"id":304,"date":"2020-07-17T20:14:58","date_gmt":"2020-07-17T19:14:58","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=304"},"modified":"2020-07-21T12:02:33","modified_gmt":"2020-07-21T11:02:33","slug":"monitoring-influxdb-telegraf-grafana","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2020\/07\/17\/monitoring-influxdb-telegraf-grafana\/","title":{"rendered":"Monitoring: InfluxDB-Telegraf-Grafana"},"content":{"rendered":"\n<p>This is something I wanted to try for some time. Normally for networks monitoring you use a NMS tool. They can be expensive, free or cheap. I have seen\/used Observium and <a href=\"https:\/\/www.librenms.org\/\">LibreNMS<\/a>. And many years ago Cacti. There are other tools that can do the job like Zabbix\/Nagios\/Icinga.<\/p>\n\n\n\n<p>But it seems time-series-databases are the new standard. They give you more flexibility as you can create queries and graph them.<\/p>\n\n\n\n<p>There are many tools out there that I dont really know like <a href=\"https:\/\/prometheus.io\/\">Prometheus<\/a>, the <a href=\"https:\/\/www.elastic.co\/what-is\/elk-stack\">elk-stack<\/a> (Elasticsearch, Logstash, and Kibana), <a href=\"https:\/\/www.influxdata.com\/\">Influxdb<\/a>, <a href=\"https:\/\/www.influxdata.com\/time-series-platform\/telegraf\/\">telegraf<\/a> and <a href=\"https:\/\/grafana.com\/\">grafana<\/a>.<\/p>\n\n\n\n<p>I decided for InfluxDB-Telegraf-Grafana stuck as I could find quickly info based on scenarios of networks.<\/p>\n\n\n\n<p>What is the rule of eachc one:<\/p>\n\n\n\n<p>Telegraf: collect data<br>InfluxDB: store data<br>Grafana: visualize<\/p>\n\n\n\n<p>My main source is again Anton&#8217;s <a href=\"https:\/\/karneliuk.com\/2019\/03\/sp-part-6-secured-monitoring-of-multivendor-service-provider-fabric-with-telegraf-influxdb-and-grafana-running-as-docker-containers-and-automated-with-ansible\/\">blog<\/a>. All credits to him.<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Environment<\/h3>\n\n\n\n<p>My network is just 3 Arista ceos containers via docker. All services will run as containers so you need docker installed. Everything is IPv4.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">InfluxDB<\/h3>\n\n\n\n<p>Installation:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/\/ Create directories\nmkdir telemetry-example\/influxdb\ncd telemetry-example\/influxdb\n\n\/\/ Get influxdb config\ndocker run --rm influxdb influxd config &gt; influxdb.conf\n\n\/\/ Create local data folder for influxdb that we will map\nmkdir data\nls -ltr\n\n\/\/ Check docker status\ndocker images\ndocker ps -a\n\n\/\/ Create docker instance for influxdb. Keep in  mind that I am giving a name to the instance\n\ndocker run -d -p 8086:8086 -p 8088:8088 --name influxdb \\\n-v $PWD\/influxdb.conf:\/etc\/influxdb\/influxdb.conf:ro \\\n-v $PWD\/data:\/var\/lib\/influxdb \\\ninfluxdb -config \/etc\/influxdb\/influxdb.conf\n\n\/\/ Verify connectivity\ncurl -i http:\/\/localhost:8086\/ping\n\n\/\/ Create database \"test\" using http-query (link below for more details)\ncurl -XPOST http:\/\/localhost:8086\/query --data-urlencode \"q=CREATE DATABASE test\"\n{\"results\":[{\"statement_id\":0}]} &lt;-- command was ok!\n\n\/\/ Create user\/pass for your db. \ncurl -XPOST http:\/\/localhost:8086\/query --data-urlencode \"q=CREATE USER xxx WITH PASSWORD 'xxx123' WITH ALL PRIVILEGES\"\n{\"results\":[{\"statement_id\":0}]} &lt;-- command was ok!\n\n\/\/ Create SSL cert for influxdb\ndocker exec -it influxdb openssl req -x509 -nodes -newkey rsa:2048 -keyout \/etc\/ssl\/influxdb-selfsigned.key -out \/etc\/ssl\/influxdb-selfsigned.crt -days 365 -subj \"\/C=GB\/ST=LDN\/L=LDN\/O=domain.com\/CN=influxdb.domain.com\"\n\n\/\/ Update influxdb.conf for SSL\ntelemetry-example\/influxdb$ vim influxdb.conf\n\u2026\nhttps-enabled = true\nhttps-certificate = \"\/etc\/ssl\/influxdb-selfsigned.crt\"\nhttps-private-key = \"\/etc\/ssl\/influxdb-selfsigned.key\"\n\u2026\n\n\/\/ Restart influxdb to take the changes\ndocker restart influxdb\n\n\/\/ Get influxdb IP for using it later\ndocker container inspect influxdb --format='{{ .NetworkSettings.IPAddress }}'\n172.17.0.2\n\n\/\/ Verify connectivity via https\ncurl -i https:\/\/localhost:8086\/ping --insecure<\/pre>\n\n\n\n<p>The verification for HTTPS was a bit more difficult because the result was always correct no matter what query I was running:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ curl -G https:\/\/localhost:8086\/query --data-urlencode \"db=test\" --data-urlencode \"q=SELECT * FROM test\" --insecure\n{\"results\":[{\"statement_id\":0}]}\n\n$ curl -XPOST 'https:\/\/localhost:8086\/query?db=test&amp;u=xxx&amp;p=xxx123' --data-urlencode 'q=SELECT * FROM test' --insecure\n{\"results\":[{\"statement_id\":0}]}\n\n$ curl -XPOST 'https:\/\/localhost:8086\/query?db=test&amp;u=xxx&amp;p=yyy1231' --data-urlencode 'q=SELECT * FROM test' --insecure\n{\"results\":[{\"statement_id\":0}]}<\/pre>\n\n\n\n<p>So I decided to see if there was cli\/shell for the influxdb (like in mysql, etc). And yes, there is <a href=\"https:\/\/docs.influxdata.com\/influxdb\/v1.8\/tools\/shell\/\">one<\/a>. Keep in  mind that you have to use &#8220;-ssl -unsafeSsl&#8221; at the same time! That confused me a lot.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ docker exec -it influxdb influx <strong>-ssl -unsafeSsl<\/strong>\nConnected to https:\/\/localhost:8086 version 1.8.1\nInfluxDB shell version: 1.8.1\n&gt; show databases\nname: databases\nname\n_internal\ntest\n&gt; use test\nUsing database test\n&gt; show series\nkey\ncpu,cpu=cpu-total,host=5f7aa2c5550e<\/pre>\n\n\n\n<p>Links about influxdb that are good for the docker creation and the http queries:<\/p>\n\n\n\n<p><a href=\"https:\/\/hub.docker.com\/_\/influxdb\">https:\/\/hub.docker.com\/_\/influxdb<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/docs.influxdata.com\/influxdb\/v1.7\/tools\/api\/#query-http-endpoin\">https:\/\/docs.influxdata.com\/influxdb\/v1.7\/tools\/api\/#query-http-endpoin<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Telegraf<\/h3>\n\n\n\n<p>I struggled with the SNMP config needed in Telegraf. The installation was fine.<\/p>\n\n\n\n<p>Links:<\/p>\n\n\n\n<p><a href=\"https:\/\/hub.docker.com\/_\/telegraf\">https:\/\/hub.docker.com\/_\/telegraf<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/docs.influxdata.com\/telegraf\/v1.14\/introduction\/getting-started\/\">https:\/\/docs.influxdata.com\/telegraf\/v1.14\/introduction\/getting-started\/<\/a><\/p>\n\n\n\n<p>Steps:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/\/ Create dir\nmkdir telemetry-example\/telegraf\ncd telemetry-example\/telegraf\n\n\/\/ Get config file to be modified\ndocker run --rm telegraf telegraf config &gt; telegraf.conf\n\n\/\/ Add the details of influxdb in telegraf.conf. As well, you need to add the devices you want to poll. In my case 172.23.0.2\/3\/4.\n\nvim telegraf.conf\n....\n<em><strong>[[outputs.influxdb]]<\/strong><\/em>\nurls = [\"https:\/\/172.17.0.2:8086\"]\ndatabase = \"test\"\nskip_database_creation = false\n## Timeout for HTTP messages.\ntimeout = \"5s\"\n## HTTP Basic Auth\nusername = \"xxx\"\npassword = \"xxx123\"\n## Use TLS but skip chain &amp; host verification\ninsecure_skip_verify = true\n# Retrieves SNMP values from remote agents\n<em><strong>[[inputs.snmp]]<\/strong><\/em>\n## Agent addresses to retrieve values from.\n## example: agents = [\"udp:\/\/127.0.0.1:161\"]\n## agents = [\"tcp:\/\/127.0.0.1:161\"]\n<code>agents = [\"udp:\/\/172.23.0.2:161\",\"udp:\/\/172.23.0.3:161\",\"udp:\/\/172.23.0.4:161\"]<\/code>\n#\n## Timeout for each request.\n<code>timeout = \"5s\"<\/code>\n#\n## SNMP version; can be 1, 2, or 3.\n<code>version = 2<\/code>\n#\n## SNMP community string.\n<code>community = \"tomas123\"<\/code>\n#\n## Number of retries to attempt.\n<code>retries = 3<\/code><\/pre>\n\n\n\n<p>This is the SNMP config I added below the SNMPv3 options in [[inputs.snmp]]<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#   ## Add fields and tables defining the variables you wish to collect.  This\n#   ## example collects the system uptime and interface variables.  Reference the\n#   ## full plugin documentation for configuration details.\n\n  &#91;&#91;inputs.snmp.field]]\n    name = \"hostname\"\n    oid = \"RFC1213-MIB::sysName.0\"\n    is_tag = true\n\n  &#91;&#91;inputs.snmp.field]]\n    name = \"uptime\"\n    oid = \"DISMAN-EVENT-MIB::sysUpTimeInstance\"\n\n  # IF-MIB::ifTable contains counters on input and output traffic as well as errors and discards.\n  &#91;&#91;inputs.snmp.table]]\n    name = \"interface\"\n    inherit_tags = &#91; \"hostname\" ]\n    oid = \"IF-MIB::ifTable\"\n\n    # Interface tag - used to identify interface in metrics database\n    &#91;&#91;inputs.snmp.table.field]]\n      name = \"ifDescr\"\n      oid = \"IF-MIB::ifDescr\"\n      is_tag = true\n\n  # IF-MIB::ifXTable contains newer High Capacity (HC) counters that do not overflow as fast for a few of the ifTable counters\n  &#91;&#91;inputs.snmp.table]]\n    name = \"interfaceX\"\n    inherit_tags = &#91; \"hostname\" ]\n    oid = \"IF-MIB::ifXTable\"\n\n    # Interface tag - used to identify interface in metrics database\n    &#91;&#91;inputs.snmp.table.field]]\n      name = \"ifDescr\"\n      oid = \"IF-MIB::ifDescr\"\n      is_tag = true\n\n  # EtherLike-MIB::dot3StatsTable contains detailed ethernet-level information about what kind of errors have been logged on an interface (such as FCS error, frame too long, etc)\n  &#91;&#91;inputs.snmp.table]]\n    name = \"interface\"\n    inherit_tags = &#91; \"hostname\" ]\n    oid = \"EtherLike-MIB::dot3StatsTable\"\n\n    # Interface tag - used to identify interface in metrics database\n    &#91;&#91;inputs.snmp.table.field]]\n      name = \"name\"\n      oid = \"IF-MIB::ifDescr\"\n      is_tag = true\n<\/code><\/pre>\n\n\n\n<p>For more info about the SNMP config in telegraf. These are good links. This is the official github <a href=\"https:\/\/github.com\/influxdata\/telegraf\">page<\/a>. And this is the page for SNMP input <a href=\"https:\/\/github.com\/influxdata\/telegraf\/tree\/master\/plugins\/inputs\/snmp\">plugin<\/a> that explain the differences between &#8220;field&#8221; and &#8220;table&#8221;.<\/p>\n\n\n\n<p>As well, the link below is <strong>really good <\/strong>too for explaining the SNMP config in telegraf:&#8221;Gathering Data via SNMP&#8221;<\/p>\n\n\n\n<p><a href=\"https:\/\/blog.networktocode.com\/post\/network_telemetry_for_snmp_devices\/\">https:\/\/blog.networktocode.com\/post\/network_telemetry_for_snmp_devices\/<\/a><\/p>\n\n\n\n<p>Start the container:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker run -d -p 8125:8125 -p 8092:8092 -p 8094:8094 --name telegraf \\\n-v $PWD\/telegraf.conf:\/etc\/telegraf\/telegraf.conf:ro \\\ntelegraf -config \/etc\/telegraf\/telegraf.conf<\/pre>\n\n\n\n<p>Check the logs:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker logs telegraf -f\n...\n2020-07-17T12:45:10Z E! [inputs.snmp] Error in plugin: initializing table interface: translating: exit status 2: MIB search path: \/root\/.snmp\/mibs:\/usr\/share\/snmp\/mibs:\/usr\/share\/snmp\/mibs\/iana:\/usr\/share\/snmp\/mibs\/ietf:\/usr\/share\/mibs\/site:\/usr\/share\/snmp\/mibs:\/usr\/share\/mibs\/iana:\/usr\/share\/mibs\/ietf:\/usr\/share\/mibs\/netsnmp\nCannot find module (EtherLike-MIB): At line 0 in (none)\nEtherLike-MIB::dot3StatsTable: Unknown Object Identifier\n...<\/pre>\n\n\n\n<p>You will see errors about not able to find the MIB files! So I used Librenms mibs. I download the project and copied the MIBS I thought I needed (arista and some other that dont belong to a vendor). As well, this is noted by Anton&#8217;s in this <a href=\"https:\/\/github.com\/influxdata\/telegraf\/issues\/4896\">link<\/a>:<\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/librenms\/librenms\/tree\/master\/mibs\">https:\/\/github.com\/librenms\/librenms\/tree\/master\/mibs<\/a><\/p>\n\n\n\n<p>In my case:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/usr\/share\/snmp\/mibs$ ls -ltr\ntotal 4672\n-rw-r--r-- 1 root root  52820 Feb  7  2019 UCD-SNMP-MIB.txt\n-rw-r--r-- 1 root root  18274 Feb  7  2019 UCD-SNMP-MIB-OLD.txt\n-rw-r--r-- 1 root root   8118 Feb  7  2019 UCD-IPFWACC-MIB.txt\n-rw-r--r-- 1 root root   6476 Feb  7  2019 UCD-IPFILTER-MIB.txt\n-rw-r--r-- 1 root root   3087 Feb  7  2019 UCD-DLMOD-MIB.txt\n-rw-r--r-- 1 root root   4965 Feb  7  2019 UCD-DISKIO-MIB.txt\n-rw-r--r-- 1 root root   2163 Feb  7  2019 UCD-DEMO-MIB.txt\n-rw-r--r-- 1 root root   5039 Feb  7  2019 NET-SNMP-VACM-MIB.txt\n-rw-r--r-- 1 root root   4814 Feb  7  2019 NET-SNMP-TC.txt\n-rw-r--r-- 1 root root   1226 Feb  7  2019 NET-SNMP-SYSTEM-MIB.txt\n-rw-r--r-- 1 root root   2504 Feb  7  2019 NET-SNMP-PERIODIC-NOTIFY-MIB.txt\n-rw-r--r-- 1 root root   3730 Feb  7  2019 NET-SNMP-PASS-MIB.txt\n-rw-r--r-- 1 root root   1215 Feb  7  2019 NET-SNMP-MONITOR-MIB.txt\n-rw-r--r-- 1 root root   2036 Feb  7  2019 NET-SNMP-MIB.txt\n-rw-r--r-- 1 root root   9326 Feb  7  2019 NET-SNMP-EXTEND-MIB.txt\n-rw-r--r-- 1 root root   9160 Feb  7  2019 NET-SNMP-EXAMPLES-MIB.txt\n-rw-r--r-- 1 root root  15901 Feb  7  2019 NET-SNMP-AGENT-MIB.txt\n-rw-r--r-- 1 root root   5931 Feb  7  2019 LM-SENSORS-MIB.txt\n-rw-r--r-- 1 root root   1913 Jul  2 04:38 GNOME-SMI.txt\n-rw-r--r-- 1 root root   5775 Jul 17 13:14 SNMPv2-TM\n-rw-r--r-- 1 root root   2501 Jul 17 13:14 SNMPv2-TC-v1\n-rw-r--r-- 1 root root  38034 Jul 17 13:14 SNMPv2-TC\n-rw-r--r-- 1 root root   1371 Jul 17 13:14 SNMPv2-SMI-v1\n-rw-r--r-- 1 root root   8924 Jul 17 13:14 SNMPv2-SMI\n-rw-r--r-- 1 root root  29305 Jul 17 13:14 SNMPv2-MIB\n-rw-r--r-- 1 root root   8263 Jul 17 13:14 SNMPv2-CONF\n-rw-r--r-- 1 root root  17177 Jul 17 13:14 INET-ADDRESS-MIB\n-rw-r--r-- 1 root root  71691 Jul 17 13:14 IF-MIB\n-rw-r--r-- 1 root root   3129 Jul 17 13:15 ARISTA-BGP4V2-TC-MIB\n-rw-r--r-- 1 root root  64691 Jul 17 13:15 ARISTA-BGP4V2-MIB\n-rw-r--r-- 1 root root   7155 Jul 17 13:15 ARISTA-VRF-MIB\n-rw-r--r-- 1 root root   1964 Jul 17 13:15 ARISTA-SMI-MIB\n-rw-r--r-- 1 root root  10901 Jul 17 13:15 ARISTA-NEXTHOP-GROUP-MIB\n-rw-r--r-- 1 root root   5826 Jul 17 13:15 ARISTA-IF-MIB\n-rw-r--r-- 1 root root   4547 Jul 17 13:15 ARISTA-GENERAL-MIB\n-rw-r--r-- 1 root root   7014 Jul 17 13:15 ARISTA-ENTITY-SENSOR-MIB\n-rw-r--r-- 1 root root  62277 Jul 17 13:21 IANA-PRINTER-MIB\n-rw-r--r-- 1 root root  36816 Jul 17 13:21 IANA-MAU-MIB\n-rw-r--r-- 1 root root   4299 Jul 17 13:21 IANA-LANGUAGE-MIB\n-rw-r--r-- 1 root root  13954 Jul 17 13:21 IANA-ITU-ALARM-TC-MIB\n-rw-r--r-- 1 root root  35628 Jul 17 13:21 IANAifType-MIB\n-rw-r--r-- 1 root root  15150 Jul 17 13:21 IANA-GMPLS-TC-MIB\n-rw-r--r-- 1 root root  10568 Jul 17 13:21 IANA-CHARSET-MIB\n-rw-r--r-- 1 root root   4743 Jul 17 13:21 IANA-ADDRESS-FAMILY-NUMBERS-MIB\n-rw-r--r-- 1 root root   3518 Jul 17 13:21 IANA-RTPROTO-MIB\n-rw-r--r-- 1 root root  13100 Jul 17 13:31 ENTITY-STATE-MIB\n-rw-r--r-- 1 root root  16248 Jul 17 13:31 ENTITY-SENSOR-MIB\n-rw-r--r-- 1 root root  59499 Jul 17 13:31 ENTITY-MIB\n-rw-r--r-- 1 root root   2114 Jul 17 13:31 BGP4V2-TC-MIB\n-rw-r--r-- 1 root root  50513 Jul 17 13:31 BGP4-MIB\n-rw-r--r-- 1 root root  96970 Jul 17 13:31 IEEE8021-CFMD8-MIB\n-rw-r--r-- 1 root root  86455 Jul 17 13:31 IEEE8021-BRIDGE-MIB\n-rw-r--r-- 1 root root 113507 Jul 17 13:31 IEEE802171-CFM-MIB\n-rw-r--r-- 1 root root   6321 Jul 17 13:31 ENTITY-STATE-TC-MIB\n-rw-r--r-- 1 root root 112096 Jul 17 13:31 IEEE802dot11-MIB\n-rw-r--r-- 1 root root  44559 Jul 17 13:31 IEEE8023-LAG-MIB\n-rw-r--r-- 1 root root  24536 Jul 17 13:31 IEEE8021-TC-MIB\n-rw-r--r-- 1 root root  62182 Jul 17 13:31 IEEE8021-SECY-MIB\n-rw-r--r-- 1 root root  96020 Jul 17 13:31 IEEE8021-Q-BRIDGE-MIB\n-rw-r--r-- 1 root root  62591 Jul 17 13:31 IEEE8021-PAE-MIB\n-rw-r--r-- 1 root root 135156 Jul 17 13:31 IEEE8021-CFM-MIB\n-rw-r--r-- 1 root root  48703 Jul 17 13:31 IPV6-MIB\n-rw-r--r-- 1 root root  15936 Jul 17 13:31 IPV6-ICMP-MIB\n-rw-r--r-- 1 root root   3768 Jul 17 13:31 IPV6-FLOW-LABEL-MIB\n-rw-r--r-- 1 root root  31323 Jul 17 13:31 IPMROUTE-STD-MIB\n-rw-r--r-- 1 root root  33626 Jul 17 13:31 IPMROUTE-MIB\n-rw-r--r-- 1 root root 186550 Jul 17 13:31 IP-MIB\n-rw-r--r-- 1 root root  46366 Jul 17 13:31 IP-FORWARD-MIB\n-rw-r--r-- 1 root root 240526 Jul 17 13:31 IEEE-802DOT17-RPR-MIB\n-rw-r--r-- 1 root root   4400 Jul 17 13:31 IPV6-UDP-MIB\n-rw-r--r-- 1 root root   7257 Jul 17 13:31 IPV6-TCP-MIB\n-rw-r--r-- 1 root root   2367 Jul 17 13:31 IPV6-TC\n-rw-r--r-- 1 root root  14758 Jul 17 13:31 IPV6-MLD-MIB\n-rw-r--r-- 1 root root  69938 Jul 17 13:31 MPLS-LSR-MIB\n-rw-r--r-- 1 root root  61017 Jul 17 13:31 MPLS-L3VPN-STD-MIB\n-rw-r--r-- 1 root root  16414 Jul 17 13:31 LLDP-V2-TC-MIB.mib\n-rw-r--r-- 1 root root  77651 Jul 17 13:31 LLDP-V2-MIB.mib\n-rw-r--r-- 1 root root  76945 Jul 17 13:31 LLDP-MIB\n-rw-r--r-- 1 root root  59110 Jul 17 13:31 LLDP-EXT-MED-MIB\n-rw-r--r-- 1 root root  30192 Jul 17 13:31 LLDP-EXT-DOT3-MIB\n-rw-r--r-- 1 root root  30182 Jul 17 13:31 LLDP-EXT-DOT1-MIB\n-rw-r--r-- 1 root root  36147 Jul 17 13:31 LLDP-EXT-DCBX-MIB\n-rw-r--r-- 1 root root 145796 Jul 17 13:31 ISIS-MIB\n-rw-r--r-- 1 root root  28564 Jul 17 13:31 TCP-MIB\n-rw-r--r-- 1 root root   1291 Jul 17 13:31 RFC-1215\n-rw-r--r-- 1 root root  79667 Jul 17 13:31 RFC1213-MIB\n-rw-r--r-- 1 root root   2866 Jul 17 13:31 RFC-1212\n-rw-r--r-- 1 root root   3067 Jul 17 13:31 RFC1155-SMI\n-rw-r--r-- 1 root root  60053 Jul 17 13:31 MPLS-VPN-MIB\n-rw-r--r-- 1 root root  95418 Jul 17 13:31 MPLS-TE-STD-MIB\n-rw-r--r-- 1 root root  55490 Jul 17 13:31 MPLS-TE-MIB\n-rw-r--r-- 1 root root  26327 Jul 17 13:31 MPLS-TC-STD-MIB\n-rw-r--r-- 1 root root  76361 Jul 17 13:31 MPLS-LSR-STD-MIB\n-rw-r--r-- 1 root root  12931 Jul 17 13:31 RFC1389-MIB\n-rw-r--r-- 1 root root  30091 Jul 17 13:31 RFC1284-MIB\n-rw-r--r-- 1 root root 147614 Jul 17 13:31 RFC1271-MIB\n-rw-r--r-- 1 root root  22342 Jul 17 13:33 SNMP-FRAMEWORK-MIB\n-rw-r--r-- 1 root root 223833 Jul 17 13:33 RMON2-MIB\n-rw-r--r-- 1 root root 127407 Jul 17 13:33 DIFFSERV-MIB\n-rw-r--r-- 1 root root 101324 Jul 17 13:34 TOKEN-RING-RMON-MIB\n-rw-r--r-- 1 root root 147822 Jul 17 13:34 RMON-MIB\n-rw-r--r-- 1 root root  26750 Jul 17 13:34 INTEGRATED-SERVICES-MIB\n-rw-r--r-- 1 root root   1863 Jul 17 13:34 DIFFSERV-DSCP-TC\n-rw-r--r-- 1 root root  34162 Jul 17 13:34 SNMP-VIEW-BASED-ACM-MIB\n-rw-r--r-- 1 root root  84133 Jul 17 13:34 Q-BRIDGE-MIB\n-rw-r--r-- 1 root root  16414 Jul 17 13:34 TRANSPORT-ADDRESS-MIB\n-rw-r--r-- 1 root root  39879 Jul 17 13:35 P-BRIDGE-MIB\n-rw-r--r-- 1 root root   4660 Jul 17 13:35 HCNUM-TC\n-rw-r--r-- 1 root root  54884 Jul 17 13:35 BRIDGE-MIB\n-rw-r--r-- 1 root root   2628 Jul 17 13:35 VPN-TC-STD-MIB\n-rw-r--r-- 1 root root  10575 Jul 17 13:35 HC-PerfHist-TC-MIB\n-rw-r--r-- 1 root root  22769 Jul 17 13:50 SNMP-TARGET-MIB\n-rw-r--r-- 1 root root  84492 Jul 17 13:54 EtherLike-MIB\n-rw-r--r-- 1 root root  68177 Jul 17 13:56 DISMAN-EXPRESSION-MIB<\/code><\/pre>\n\n\n\n<p>Once you have the MIB files, you need to copy across to the telegraf container:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker cp \/usr\/share\/snmp\/mibs\/. telegraf:\/usr\/share\/snmp\/mibs\/\ndocker restart telegraf<\/pre>\n\n\n\n<p>These two links helped me a lot to troubleshoot and understand snmpwalk:<\/p>\n\n\n\n<p><a href=\"https:\/\/www.dev-eth0.de\/2016\/12\/06\/grafana_snmp\/\">https:\/\/www.dev-eth0.de\/2016\/12\/06\/grafana_snmp\/<\/a><\/p>\n\n\n\n<figure class=\"wp-block-embed-wordpress wp-block-embed is-type-wp-embed is-provider-comparitech\"><div class=\"wp-block-embed__wrapper\">\n<blockquote class=\"wp-embedded-content\" data-secret=\"VDPRh2ww6w\"><a href=\"https:\/\/www.comparitech.com\/net-admin\/snmpwalk-examples-windows-linux\/\">snmpwalk Examples &#038; Commands for Windows and Linux<\/a><\/blockquote><iframe loading=\"lazy\" class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;snmpwalk Examples &#038; Commands for Windows and Linux&#8221; &#8212; Comparitech\" src=\"https:\/\/www.comparitech.com\/net-admin\/snmpwalk-examples-windows-linux\/embed\/#?secret=VDPRh2ww6w\" data-secret=\"VDPRh2ww6w\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>If you dont want to see the OIDS in snmpwalk, you need to load all your MIBS with -m ALL<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><em>\/usr\/share\/snmp\/mibs$ snmpwalk -Os -v2c -c community <strong>-m ALL<\/strong> 172.23.0.2 | head -10<\/em>\nsysDescr.0 = STRING: Linux r01 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64\nsysObjectID.0 = OID: aristaProducts.2600\nsysUpTimeInstance = Timeticks: (2503676) 6:57:16.76\nsysContact.0 = STRING:\n<strong>sysName.0 = STRING: r01<\/strong>\nsysLocation.0 = STRING: ceoslab\nsysServices.0 = INTEGER: 14\nsysORLastChange.0 = Timeticks: (35427) 0:05:54.27\nsysORID.1 = OID: tcpMIB\nsysORID.2 = OID: mib-2.50\n\n<strong>$ snmpwalk -v2c -c community -m ALL 172.23.0.2 | head -10<\/strong>\nSNMPv2-MIB::sysDescr.0 = STRING: Linux r01 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64\nSNMPv2-MIB::sysObjectID.0 = OID: ARISTA-SMI-MIB::aristaProducts.2600\nDISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (2535736) 7:02:37.36\nSNMPv2-MIB::sysContact.0 = STRING:\nSNMPv2-MIB::sysName.0 = STRING: r01\nSNMPv2-MIB::sysLocation.0 = STRING: ceoslab\nSNMPv2-MIB::sysServices.0 = INTEGER: 14\nSNMPv2-MIB::sysORLastChange.0 = Timeticks: (35427) 0:05:54.27\nSNMPv2-MIB::sysORID.1 = OID: TCP-MIB::tcpMIB\nSNMPv2-MIB::sysORID.2 = OID: SNMPv2-SMI::mib-2.50\n\n<em>\/usr\/share\/snmp\/mibs$ snmpwalk -Os -v2c -c community 172.23.0.2 | head -10<\/em>\niso.3.6.1.2.1.1.1.0 = STRING: \"Linux r01 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64\"\niso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.30065.1.2600\niso.3.6.1.2.1.1.3.0 = Timeticks: (2505381) 6:57:33.81\niso.3.6.1.2.1.1.4.0 = \"\"\n<strong>iso.3.6.1.2.1.1.5.0 = STRING: \"r01\"<\/strong>\niso.3.6.1.2.1.1.6.0 = STRING: \"ceoslab\"\niso.3.6.1.2.1.1.7.0 = INTEGER: 14\niso.3.6.1.2.1.1.8.0 = Timeticks: (35427) 0:05:54.27\niso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.2.1.49\niso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.2.1.50<\/pre>\n\n\n\n<p>And if you want to verify that telegraf is capable to use the MIBS:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">docker exec -it telegraf snmpwalk -Os -v2c -c community -m ALL 172.23.0.2 | head -10<\/pre>\n\n\n\n<p>Now, you can check if telegraf is updating influxdb. If there is output, it is good!<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ curl -G 'https:\/\/localhost:8086\/query?db=test&amp;pretty=true&amp;u=xxx&amp;p=xxx123' --data-urlencode \"q=SELECT * FROM interfaceX limit 2\" --insecure\n{\n\"results\": [\n{\n\"statement_id\": 0,\n\"series\": [\n{\n\"name\": \"interfaceX\",\n\"columns\": [\n\"time\",\n\"agent_host\",\n\"host\",\n\"hostname\",\n\"ifAlias\",\n\"ifConnectorPresent\",\n\"ifCounterDiscontinuityTime\",\n\"ifDescr\",\n\"ifHCInBroadcastPkts\",\n\"ifHCInMulticastPkts\",\n\"ifHCInOctets\",\n\"ifHCInUcastPkts\",\n\"ifHCOutBroadcastPkts\",\n\"ifHCOutMulticastPkts\",\n\"ifHCOutOctets\",\n\"ifHCOutUcastPkts\",\n\"ifHighSpeed\",\n\"ifInBroadcastPkts\",\n\"ifInMulticastPkts\",\n\"ifLinkUpDownTrapEnable\",\n\"ifName\",\n\"ifOutBroadcastPkts\",\n\"ifOutMulticastPkts\",\n\"ifPromiscuousMode\",\n\"name\"\n],\n\"values\": [\n[\n\"2020-07-17T13:00:10Z\",\n\"172.23.0.2\",\n\"6778bdf4ea85\",\n\"r01\",\nnull,\n1,\n0,\n\"Ethernet1\",\n0,\n2118,\n2013125,\n7824,\n0,\n0,\n0,\n0,\n0,\n0,\n2118,\n1,\n\"Ethernet1\",\n0,\n0,\n2,\nnull\n],\n[\n\"2020-07-17T13:00:10Z\",\n\"172.23.0.2\",\n\"6778bdf4ea85\",\n\"r01\",\n\"CORE Loopback\",\n2,\n0,\n\"Loopback1\",\n0,\n0,\n0,\n0,\n0,\n0,\n0,\n0,\n0,\n0,\n0,\n1,\n\"Loopback1\",\n0,\n0,\n2,\nnull\n]\n]\n}\n]\n}\n]\n}<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Grafana<\/h3>\n\n\n\n<p>I have seen Grafana before but I have never used it so the configuration on queries was a bit of a challenge but I was lucky and I found very good blogs for that. The installation process is ok:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/\/ Create folder for grafana and data\nmkdir -p telemetry-example\/grafana\/data\ncd telemetry-example\/grafana\n\n\/\/ Create docker instance\ndocker run -d -p 3000:3000 --name grafana \\\n--user root \\\n-v $PWD\/data:\/var\/lib\/grafana \\\ngrafana\/grafana\n\n\/\/ Create SSL cert for grafana\ndocker exec -it grafana openssl req -x509 -nodes -newkey rsa:2048 -keyout \/etc\/ssl\/grafana-selfsigned.key -out \/etc\/ssl\/grafana-selfsigned.crt -days 365 -subj \"\/C=GB\/ST=LDN\/L=LDN\/O=domain.com\/CN=grafana.domain.com\"\n\n\/\/ Copy grafana config so we can update it\ndocker cp grafana:\/etc\/grafana\/grafana.ini grafana.ini\n\n\/\/ Update grafana config with SSL\nvim grafana.ini\n############################## Server\n[server]\n# Protocol (http, https, h2, socket)\nprotocol = https\n\u2026\n# https certs &amp; key file\ncert_file = \/etc\/ssl\/grafana-selfsigned.crt\ncert_key = \/etc\/ssl\/grafana-selfsigned.key\n\n\/\/ Copy back the config to the container and restart\ndocker cp grafana.ini grafana:\/etc\/grafana\/grafana.ini\ndocker container restart grafana<\/pre>\n\n\n\n<p>Now you can open in your browser to grafana &#8220;https:\/\/0.0.0.0:3000\/ &#8221; using admin\/admin<\/p>\n\n\n\n<p>You need to add a data source that is our influxdb container. So you need to pick up the &#8220;influxdb&#8221; type and fill the values as per below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"637\" height=\"1024\" src=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-config-influxdb-637x1024.png\" alt=\"\" class=\"wp-image-307\" srcset=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-config-influxdb-637x1024.png 637w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-config-influxdb-186x300.png 186w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-config-influxdb-768x1236.png 768w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-config-influxdb.png 823w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 984px) 61vw, (max-width: 1362px) 45vw, 600px\" \/><\/figure>\n\n\n\n<p>Now, you need to create a dashboard with panel.<\/p>\n\n\n\n<p>Links that I reviewed for creating the dasbord <\/p>\n\n\n\n<p>For creating  a panel. The link below was the <strong>best<\/strong> on section &#8220;Interface Throughput&#8221;. Big thanks to the author.<\/p>\n\n\n\n<p><a href=\"https:\/\/lkhill.com\/telegraf-influx-grafana-network-stats\/\">https:\/\/lkhill.com\/telegraf-influx-grafana-network-stats\/<\/a><\/p>\n\n\n\n<p>This is my query for checking all details:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"668\" src=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query-1024x668.png\" alt=\"\" class=\"wp-image-308\" srcset=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query-1024x668.png 1024w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query-300x196.png 300w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query-768x501.png 768w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query-1536x1002.png 1536w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query-1200x783.png 1200w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-creat-query.png 2017w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/figure>\n\n\n\n<p>And this is my final dashboard.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"884\" height=\"1024\" src=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-dashboard-884x1024.png\" alt=\"\" class=\"wp-image-309\" srcset=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-dashboard-884x1024.png 884w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-dashboard-259x300.png 259w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-dashboard-768x889.png 768w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/07\/grafana-dashboard.png 1070w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">SNMP Config<\/h3>\n\n\n\n<p>BTW, you need to config SNMP in the switches so telegraf can poll it:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">snmp-server location ceoslab\nsnmp-server community xxx123 ro\nsnmp-server host 172.17.0.1 version 2c xxx123<\/pre>\n\n\n\n<p>In my case, the stack of containers Influx-Telegraf-Grafana are running on the default bridge. Each container has its own IP but as the Arista containers are in the different docker network, it needs to &#8220;route&#8221; so the IP of telegraf container will be NAT-ed to 172.17.0.1 from the switches point of view.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Next<\/h3>\n\n\n\n<p>I would like to manage all this process via Ansible&#8230; Something like this.. but will take me time<\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/akarneliuk\/service_provider_fabric\/tree\/master\/ansible\/roles\/cloud_monitoring\/tasks\">https:\/\/github.com\/akarneliuk\/service_provider_fabric\/tree\/master\/ansible\/roles\/cloud_monitoring\/tasks<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Notes<\/h3>\n\n\n\n<p>As usual, I have struggled but I have learned a lot and at the end things are working. I am happy with that.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is something I wanted to try for some time. Normally for networks monitoring you use a NMS tool. They can be expensive, free or cheap. I have seen\/used Observium and LibreNMS. And many years ago Cacti. There are other tools that can do the job like Zabbix\/Nagios\/Icinga. But it seems time-series-databases are the new &hellip; <a href=\"https:\/\/blog.thomarite.uk\/index.php\/2020\/07\/17\/monitoring-influxdb-telegraf-grafana\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Monitoring: InfluxDB-Telegraf-Grafana&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,22],"tags":[],"class_list":["post-304","post","type-post","status-publish","format-standard","hentry","category-unix","category-monitoring"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/304","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=304"}],"version-history":[{"count":5,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/304\/revisions"}],"predecessor-version":[{"id":318,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/304\/revisions\/318"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}