{"id":345,"date":"2020-08-12T17:51:30","date_gmt":"2020-08-12T16:51:30","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=345"},"modified":"2020-08-13T10:34:44","modified_gmt":"2020-08-13T09:34:44","slug":"streaming-veos-telemetry","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2020\/08\/12\/streaming-veos-telemetry\/","title":{"rendered":"Streaming vEOS Telemetry"},"content":{"rendered":"\n

One thing I wanted to get my hands dirty is telemetry. I found this blog post<\/a> from Arista and I have tried to use it for vEOS.<\/p>\n\n\n\n

As per the blog, I had to install go and docker in my VM (debian) on GCP.<\/p>\n\n\n\n

Installed docker via aptitude:<\/p>\n\n\n\n

# aptitude install docker.io\n# aptitude install docker-compose\n# aptitude install telnet<\/pre>\n\n\n\n
Installed golang via updating .bashrc<\/p>\n\n\n\n
########################\n# Go configuration\n########################\n#\n# git clone -b v0.0.4 https:\/\/github.com\/wfarr\/goenv.git $HOME\/.goenv\nif [ ! -d \"$HOME\/.goenv\" ]; then\n    git clone https:\/\/github.com\/syndbg\/goenv.git $HOME\/.goenv\nfi\n#export GOPATH=\"$HOME\/storage\/golang\/go\"\n#export GOBIN=\"$HOME\/storage\/golang\/go\/bin\"\n#export PATH=\"$GOPATH\/bin:$PATH\"\nif [ -d \"$HOME\/.goenv\"   ]\nthen\n    export GOENV_ROOT=\"$HOME\/.goenv\"\n    export PATH=\"$GOENV_ROOT\/bin:$PATH\"\n    if  type \"goenv\" &> \/dev\/null; then\n        eval \"$(goenv init -)\"\n        # Add the version to my prompt\n        __goversion (){\n            if  type \"goenv\" &> \/dev\/null; then\n                goenv_go_version=$(goenv version | sed -e 's\/ .*\/\/')\n                printf $goenv_go_version\n            fi\n        }\n        export PS1=\"go:\\$(__goversion)|$PS1\"\n        export PATH=\"$GOROOT\/bin:$PATH\"\n        export PATH=\"$PATH:$GOPATH\/bin\"\n    fi\nfi\n\n################## End GoLang #####################<\/code><\/pre>\n\n\n\n
Then started a new bash session to trigger the installation of goenv and then install a go version<\/p>\n\n\n\n
$ goenv install 1.14.6\n$ goenv global 1.14.6<\/pre>\n\n\n\n
Now we can start the docker containers for influx and grafana:<\/p>\n\n\n\n
mkdir telemetry\ncd telemetry\nmkdir influxdb_data\nmkdir grafana_data\n\ncat docker-compose.yaml \nversion: \"3\"\n\nservices:\n influxdb:\n   container_name: influxdb-tele\n   environment:\n     INFLUXDB_DB: grpc\n     INFLUXDB_ADMIN_USER: \"admin\"\n     INFLUXDB_ADMIN_PASSWORD: \"arista\"\n     INFLUXDB_USER: tac\n     INFLUXDB_USER_PASSWORD: arista\n     INFLUXDB_RETENTION_ENABLED: \"false\"\n     INFLUXDB_OPENTSDB_0_ENABLED: \"true\"\n     INFLUXDB_OPENTSDB_BIND_ADDRESS: \":4242\"\n     INFLUXDB_OPENTSDB_DATABASE: \"grpc\"\n   ports:\n     - '8086:8086'\n     - '4242:4242'\n     - '8083:8083'\n   networks:\n     - monitoring\n   volumes:\n     - influxdb_data:\/var\/lib\/influxdb\n   command:\n     - '-config'\n     - '\/etc\/influxdb\/influxdb.conf'\n   image: influxdb:latest\n   restart: always\n\n grafana:\n   container_name: grafana-tele\n   environment:\n     GF_SECURITY_ADMIN_USER: admin\n     GF_SECURITY_ADMIN_PASSWORD: arista\n   ports:\n     - '3000:3000'\n   networks:\n     - monitoring\n   volumes:\n     - grafana_data:\/var\/lib\/grafana\n   image: grafana\/grafana\n   restart: always\n\nnetworks:\n monitoring:\n\nvolumes:\n influxdb_data: {}\n grafana_data: {}\n<\/code><\/pre>\n\n\n\n
Now we should be able to start the containers:<\/p>\n\n\n\n
sudo docker-compose up -d    \/\/ start containers\nsudo docker-compose down -v  \/\/ for stopping containers\n\nsudo docker ps -a\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\ncad339ead2ee influxdb:latest \"\/entrypoint.sh -con\u2026\" 5 hours ago Up 5 hours 0.0.0.0:4242->4242\/tcp, 0.0.0.0:8083->8083\/tcp, 0.0.0.0:8086->8086\/tcp influxdb-tele\nef88acc47ee3 grafana\/grafana \"\/run.sh\" 5 hours ago Up 5 hours 0.0.0.0:3000->3000\/tcp grafana-tele\n\nsudo docker network ls\nNETWORK ID NAME DRIVER SCOPE\nfe19e7876636 bridge bridge local\n0a8770578f3f host host local\n6e128a7682f1 none null local\n3d27d0ed3ab3 telemetry_monitoring bridge local\n\nsudo docker network inspect telemetry_monitoring\n[\n{\n\"Name\": \"telemetry_monitoring\",\n\"Id\": \"3d27d0ed3ab3b0530441206a128d849434a540f8e5a2c109ee368b01052ed418\",\n\"Created\": \"2020-08-12T11:22:03.05783331Z\",\n\"Scope\": \"local\",\n\"Driver\": \"bridge\",\n\"EnableIPv6\": false,\n\"IPAM\": {\n\"Driver\": \"default\",\n\"Options\": null,\n\"Config\": [\n{\n\"Subnet\": \"172.18.0.0\/16\",\n\"Gateway\": \"172.18.0.1\"\n}\n]\n},\n\"Internal\": false,\n\"Attachable\": true,\n\"Ingress\": false,\n\"ConfigFrom\": {\n\"Network\": \"\"\n},\n\"ConfigOnly\": false,\n\"Containers\": {\n\"cad339ead2eeb0b479bd6aa024cb2150fb1643a0a4a59e7729bb5ddf088eba19\": {\n\"Name\": \"influxdb-tele\",\n\"EndpointID\": \"e3c7f853766ed8afe6617c8fac358b3302de41f8aeab53d429ffd1a28b6df668\",\n\"MacAddress\": \"02:42:ac:12:00:03\",\n\"IPv4Address\": \"172.18.0.3\/16\",\n\"IPv6Address\": \"\"\n},\n\"ef88acc47ee30667768c5af9bbd70b95903d3690c4d80b83ba774b298665d15d\": {\n\"Name\": \"grafana-tele\",\n\"EndpointID\": \"3fe2b424cbb66a93e9e06f4bcc2e7353a0b40b2d56777c8fee8726c96c97229a\",\n\"MacAddress\": \"02:42:ac:12:00:02\",\n\"IPv4Address\": \"172.18.0.2\/16\",\n\"IPv6Address\": \"\"\n}\n},\n\"Options\": {},\n\"Labels\": {\n\"com.docker.compose.network\": \"monitoring\",\n\"com.docker.compose.project\": \"telemetry\"\n}\n}\n]<\/pre>\n\n\n\n
Now we have to generate the octsdb binary and copy it to the switches as per instructions<\/p>\n\n\n\n
$ go get github.com\/aristanetworks\/goarista\/cmd\/octsdb\n$ cd $GOPATH\/src\/github.com\/aristanetworks\/goarista\/cmd\/octsdb\n\n$ GOOS=linux GOARCH=386 go build   \/\/ I used this one\n$ GOOS=linux GOARCH=amd64 go build \/\/ if you use EOS 64 Bits\n\n$ scp octsdb user@SWITCH_IP:\/mnt\/flash\/<\/pre>\n\n\n\n
An important thing is the configuration file for octsdb.  I struggled trying to get a config file that provided me CPU and interface counters.  All the examples are based on hardware platforms but I am using containers\/VMs. But using this other blog post<\/a>, I worked out the path for the interfaces in vEOS. <\/p>\n\n\n\n
This is what I see in vEOS:<\/p>\n\n\n\n
bash-4.2# curl localhost:6060\/rest\/Smash\/counters\/ethIntf\/EtbaDut\/current\n{\n    \"counter\": {\n        \"Ethernet1\": {\n            \"counterRefreshTime\": 0,\n            \"ethStatistics\": {\n...\n\nbash-4.2# curl localhost:6060\/rest\/Kernel\/proc\/cpu\/utilization\/cpu\/0\n{\n    \"idle\": 293338,\n    \"name\": \"0\",\n    \"nice\": 2965,\n    \"system\": 1157399,\n    \"user\": 353004,\n    \"util\": 100\n}\n<\/code><\/pre>\n\n\n\n
And this is what I see in cEOS. It seems this is not functional:<\/p>\n\n\n\n
bash-4.2# curl localhost:6060\/rest\/Smash\/counters\/ethIntf\ncurl: (7) Failed to connect to localhost port 6060: Connection refused\nbash-4.2# \nbash-4.2# \nbash-4.2# curl localhost:6060\/rest\/Kernel\/proc\/cpu\/utilization\/cpu\/0\ncurl: (7) Failed to connect to localhost port 6060: Connection refused\nbash-4.2# \n<\/code><\/pre>\n\n\n\n
This is the file I have used and pushed to the switches:<\/p>\n\n\n\n
$ cat veos4.23.json \n{\n\t\"comment\": \"This is a sample configuration for vEOS 4.23\",\n\t\"subscriptions\": [\n\t\t\"\/Smash\/counters\/ethIntf\",\n\t\t\"\/Kernel\/proc\/cpu\/utilization\"\n\t],\n\t\"metricPrefix\": \"eos\",\n\t\"metrics\": {\n\t\t\"intfCounter\": {\n\t\t\t\"path\": \"\/Smash\/counters\/ethIntf\/EtbaDut\/current\/(counter)\/(?P<intf>.+)\/statistics\/(?P<direction>(?:in|out))(Octets|Errors|Discards)\"\n\t\t},\n\t\t\"intfPktCounter\": {\n\t\t\t\"path\": \"\/Smash\/counters\/ethIntf\/EtbaDut\/current\/(counter)\/(?P<intf>.+)\/statistics\/(?P<direction>(?:in|out))(?P<type>(?:Ucast|Multicast|Broadcast))(Pkt)\"\n\t\t},\n                \"totalCpu\": {\n                        \"path\": \"\/Kernel\/proc\/(cpu)\/(utilization)\/(total)\/(?P<type>.+)\"\n                },\n                \"coreCpu\": {\n                        \"path\": \"\/Kernel\/proc\/(cpu)\/(utilization)\/(.+)\/(?P<type>.+)\"\n                }\n\t}\n}\n$ scp veos4.23.json r1:\/mnt\/flash\n<\/code><\/pre>\n\n\n\n
Now you have to configure the switch to generate and send the data. In my case, I am using the MGMT vrf.<\/p>\n\n\n\n
!\ndaemon TerminAttr\n   exec \/usr\/bin\/TerminAttr -disableaaa -grpcaddr MGMT\/0.0.0.0:6042\n   no shutdown\n!\ndaemon octsdb\n   exec \/sbin\/ip netns exec ns-MGMT \/mnt\/flash\/octsdb -addr 192.168.249.4:6042 -config \/mnt\/flash\/veos4.23.json -tsdb 10.128.0.4:4242\n   no shutdown\n!\n<\/code><\/pre>\n\n\n\n
TermiAttr it is listening on 0.0.0.0:6042. “octsdb” is using the mgmt IP 192.168.249.4 (that is in MGMT vrf) to connect to the influxdb container that is running in 10.128.0.4:4242<\/p>\n\n\n\n
Verify that things are running:<\/p>\n\n\n\n
# From the switch\nshow agent octsdb log\n\n# From InfluxDB container\n$ sudo docker exec -it influxdb-tele bash\nroot@cad339ead2ee:\/# influx -precision 'rfc3339'\nConnected to http:\/\/localhost:8086 version 1.8.1\nInfluxDB shell version: 1.8.1\n> show databases\nname: databases\nname\n----\ngrpc\n_internal\n> use grpc\nUsing database grpc\n> show measurements\nname: measurements\nname\n----\neos.corecpu.cpu.utilization._counts\neos.corecpu.cpu.utilization.cpu.0\neos.corecpu.cpu.utilization.total\neos.intfcounter.counter.discards\neos.intfcounter.counter.errors\neos.intfcounter.counter.octets\neos.intfpktcounter.counter.pkt\neos.totalcpu.cpu.utilization.total\n> exit\nroot@cad339ead2ee:\/# exit\n<\/code><\/pre>\n\n\n\n
Now we need to configure grafana. So first we create a connection to influxdb. Again, I struggled with the URL. Influx and grafana are two containers running in the same host. I was using initially localhost and it was failing. At the end I had to find out the IP assigned to the influxdb container and use it.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now you can create a dashboard with panels.<\/p>\n\n\n\n
For octet counters:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
For packet types:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
For CPU usage:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
And this is the dashboard:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Keep in mind that I am not 100% sure my grafana panels are correct for CPU and Counters (PktCounter makes sense)<\/p>\n\n\n\n
At some point, I want to try telemetry for LANZ<\/a> via this telerista<\/a> plugin.<\/p>\n","protected":false},"excerpt":{"rendered":"
One thing I wanted to get my hands dirty is telemetry. I found this blog post from Arista and I have tried to use it for vEOS. As per the blog, I had to install go and docker in my VM (debian) on GCP. Installed docker via aptitude: # aptitude install docker.io # aptitude install … Continue reading “Streaming vEOS Telemetry”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[],"class_list":["post-345","post","type-post","status-publish","format-standard","hentry","category-monitoring"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=345"}],"version-history":[{"count":2,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/345\/revisions"}],"predecessor-version":[{"id":352,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/345\/revisions\/352"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}