{"id":469,"date":"2020-11-08T11:44:38","date_gmt":"2020-11-08T11:44:38","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=469"},"modified":"2023-05-29T18:19:22","modified_gmt":"2023-05-29T17:19:22","slug":"gnmi-ssl-p2","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2020\/11\/08\/gnmi-ssl-p2\/","title":{"rendered":"gnmi-ssl-p2"},"content":{"rendered":"\n

I was already playing with gNMI and protobuf a couple of months ago<\/a>. But this week I received a summary from the last NANOG80<\/a> meeting and there was a presentation<\/a> about it. Great job from Colin!<\/p>\n\n\n\n

So I decided to give it a go as the demo was based on docker and I have already my Arista lab in cEOS and vEOS as targets.<\/p>\n\n\n\n

I started my 3node-ring cEOS lab with docker-topo<\/a><\/p>\n\n\n\n

ceos-testing\/topology master$ docker-topo --create 3-node-simple.yml\nINFO:main<\/strong>:Version 2 requires sudo. Restarting script with sudo\n[sudo] password for xxx:\nINFO:main<\/strong>:\nalias r01='docker exec -it 3node_r01 Cli'\nalias r02='docker exec -it 3node_r02 Cli'\nalias r03='docker exec -it 3node_r03 Cli'\nINFO:main<\/strong>:All devices started successfully<\/pre>\n\n\n\n
Checked they were up:<\/p>\n\n\n\n
$ docker ps -a\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\n4160cc354ba2 ceos-lab:4.23.3M \"\/sbin\/init systemd.\u2026\" 7 minutes ago Up 7 minutes 0.0.0.0:2002->22\/tcp, 0.0.0.0:9002->443\/tcp 3node_r03\n122f72fb25bd ceos-lab:4.23.3M \"\/sbin\/init systemd.\u2026\" 7 minutes ago Up 7 minutes 0.0.0.0:2001->22\/tcp, 0.0.0.0:9001->443\/tcp 3node_r02\n68cf8ca39130 ceos-lab:4.23.3M \"\/sbin\/init systemd.\u2026\" 7 minutes ago Up 7 minutes 0.0.0.0:2000->22\/tcp, 0.0.0.0:9000->443\/tcp 3node_r01<\/pre>\n\n\n\n
And then, check I had gnmi config in r01:<\/p>\n\n\n\n
!\nmanagement api gnmi\ntransport grpc GRPC\nport 3333\n!<\/pre>\n\n\n\n
Need to find the IP of r01 in “3node_net-0” as the one used for management. I have had so many times hit this issue,…<\/p>\n\n\n\n
$ docker inspect 3node_r01\n...\n\"Networks\": {<\/code>\n \"3node_net-0\": {<\/code>\n \"IPAMConfig\": null, <\/code>\n \"Links\": null,<\/code>\n \"Aliases\": [ \"68cf8ca39130\" ],<\/code>\n \"NetworkID\": \"d3f72e7473228488f668aa3ed65b6ea94e1c5c9553f93cf0f641c3d4af644e2e\", \"EndpointID\": \"bca584040e71a826ef25b8360d92881dad407ff976eff65a38722fd36e9fc873\", \"Gateway\": \"172.20.0.1\", <\/code>\n\"IPAddress\": \"172.20.0.2\",<\/code>\n....<\/pre>\n\n\n\n
Now, I cloned the repo<\/a> and followed the instructions<\/a>\/video. Copied targets.jon and updated it with my r01 device details:<\/p>\n\n\n\n
~\/storage\/technology\/gnmi-gateway release$ cat examples\/gnmi-prometheus\/targets.json \n{\n  \"request\": {\n    \"default\": {\n      \"subscribe\": {\n        \"prefix\": {\n        },\n        \"subscription\": [\n          {\n            \"path\": {\n              \"elem\": [\n                {\n                  \"name\": \"interfaces\"\n                }\n              ]\n            }\n          }\n        ]\n      }\n    }\n  },\n  \"target\": {\n    \"r01\": {\n      \"addresses\": [\n        \"172.20.0.2:3333\"\n      ],\n      \"credentials\": {\n        \"username\": \"xxx\",\n        \"password\": \"xxx\"\n      },\n      \"request\": \"default\",\n      \"meta\": {\n        \"NoTLS\": \"yes\"\n      }\n    }\n  }\n}<\/code><\/pre>\n\n\n\n
Carrying out with the instructions<\/a>, build docker gnmi-gateway, docker bridge and run docker gnmi-gateway built earlier.<\/p>\n\n\n\n
go:1.14.6|py:3.7.3|tomas@athens:~\/storage\/technology\/gnmi-gateway release$ docker run \\\n-it --rm \\\n-p 59100:59100 \\\n-v $(pwd)\/examples\/gnmi-prometheus\/targets.json:\/opt\/gnmi-gateway\/targets.json \\\n--name gnmi-gateway-01 \\\n--network gnmi-net \\\ngnmi-gateway:latest\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Starting GNMI Gateway.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Clustering is NOT enabled. No locking or cluster coordination will happen.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Starting connection manager.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Starting gNMI server on 0.0.0.0:9339.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Starting Prometheus exporter.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Connection manager received a target control message: 1 inserts 0 removes\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Initializing target r01 ([172.27.0.2:3333]) map[NoTLS:yes].\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Target r01: Connecting\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Target r01: Subscribing\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:28Z\",\"message\":\"Starting Prometheus HTTP server.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:38Z\",\"message\":\"Target r01: Disconnected\"}\nE1107 16:54:38.382032 1 reconnect.go:114] client.Subscribe (target \"r01\") failed: client \"gnmi\" : client \"gnmi\" : Dialer(172.27.0.2:3333, 10s): context deadline exceeded; reconnecting in 552.330144ms\n{\"level\":\"info\",\"time\":\"2020-11-07T16:54:48Z\",\"message\":\"Target r01: Disconnected\"}\nE1107 16:54:48.935965 1 reconnect.go:114] client.Subscribe (target \"r01\") failed: client \"gnmi\" : client \"gnmi\" : Dialer(172.27.0.2:3333, 10s): context deadline exceeded; reconnecting in 1.080381816s<\/pre>\n\n\n\n
bash-4.2# tcpdump -i any tcp port 3333 -nnn\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode\nlistening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes\n17:07:57.621011 In 02:42:7c:61:10:40 ethertype IPv4 (0x0800), length 76: 172.27.0.1.43644 > 172.27.0.2.3333: Flags [S], seq 557316949, win 64240, options [mss 1460,sackOK,TS val 3219811744 ecr 0,nop,wscale 7], length 0\n17:07:57.621069 Out 02:42:ac:1b:00:02 ethertype IPv4 (0x0800), length 76: 172.27.0.2.3333 > 172.27.0.1.43644: Flags [S.], seq 243944609, ack 557316950, win 65160, options [mss 1460,sackOK,TS val 1828853442 ecr 3219811744,nop,wscale 7], length 0\n17:07:57.621124 In 02:42:7c:61:10:40 ethertype IPv4 (0x0800), length 68: 172.27.0.1.43644 > 172.27.0.2.3333: Flags [.], ack 1, win 502, options [nop,nop,TS val 3219811744 ecr 1828853442], length 0\n17:07:57.621348 Out 02:42:ac:1b:00:02 ethertype IPv4 (0x0800), length 89: 172.27.0.2.3333 > 172.27.0.1.43644: Flags [P.], seq 1:22, ack 1, win 510, options [nop,nop,TS val 1828853442 ecr 3219811744], length 21\n17:07:57.621409 In 02:42:7c:61:10:40 ethertype IPv4 (0x0800), length 68: 172.27.0.1.43644 > 172.27.0.2.3333: Flags [.], ack 22, win 502, options [nop,nop,TS val 3219811744 ecr 1828853442], length 0\n17:07:57.621492 In 02:42:7c:61:10:40 ethertype IPv4 (0x0800), length 320: 172.27.0.1.43644 > 172.27.0.2.3333: Flags [P.], seq 1:253, ack 22, win 502, options [nop,nop,TS val 3219811744 ecr 1828853442], length 252\n17:07:57.621509 Out 02:42:ac:1b:00:02 ethertype IPv4 (0x0800), length 68: 172.27.0.2.3333 > 172.27.0.1.43644: Flags [.], ack 253, win 509, options [nop,nop,TS val 1828853442 ecr 3219811744], length 0\n17:07:57.621586 In 02:42:7c:61:10:40 ethertype IPv4 (0x0800), length 68: 172.27.0.1.43644 > 172.27.0.2.3333: Flags [F.], seq 253, ack 22, win 502, options [nop,nop,TS val 3219811744 ecr 1828853442], length 0\n17:07:57.621904 Out 02:42:ac:1b:00:02 ethertype IPv4 (0x0800), length 68: 172.27.0.2.3333 > 172.27.0.1.43644: Flags [R.], seq 22, ack 254, win 509, options [nop,nop,TS val 1828853443 ecr 3219811744], length 0<\/pre>\n\n\n\n
Ok, the container is created and seems running but the gnmi-gateway can’t connect to my cEOS r01….<\/p>\n\n\n\n
First thing, I had to check iptables. It is not the first time that when playing with docker and building different environments (vEOS vs gnmi-gateway) with different docker commands, iptables may be not configured properly.<\/p>\n\n\n\n
And it was the case again:<\/p>\n\n\n\n
# iptables -t filter -S DOCKER-ISOLATION-STAGE-1\nWarning: iptables-legacy tables present, use iptables-legacy to see them\n-N DOCKER-ISOLATION-STAGE-1\n-A DOCKER-ISOLATION-STAGE-1 -i br-43481af25965 ! -o br-43481af25965 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -j ACCEPT\n-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-94c1e813ad6f ! -o br-94c1e813ad6f -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-4bd17cfa19a8 ! -o br-4bd17cfa19a8 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-13ab2b6a0d1d ! -o br-13ab2b6a0d1d -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-121978ca0282 ! -o br-121978ca0282 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-00db5844bbb0 ! -o br-00db5844bbb0 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -j RETURN<\/pre>\n\n\n\n
So I moved the new docker bridge network for gnmi-gateway after “ACCEPT” and solved.<\/p>\n\n\n\n
# iptables -t filter -D DOCKER-ISOLATION-STAGE-1 -j ACCEPT\n# iptables -t filter -I DOCKER-ISOLATION-STAGE-1 -j ACCEPT\n#\n# iptables -t filter -S DOCKER-ISOLATION-STAGE-1\nWarning: iptables-legacy tables present, use iptables-legacy to see them\n-N DOCKER-ISOLATION-STAGE-1\n-A DOCKER-ISOLATION-STAGE-1 -j ACCEPT\n-A DOCKER-ISOLATION-STAGE-1 -i br-43481af25965 ! -o br-43481af25965 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-94c1e813ad6f ! -o br-94c1e813ad6f -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-4bd17cfa19a8 ! -o br-4bd17cfa19a8 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-13ab2b6a0d1d ! -o br-13ab2b6a0d1d -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-121978ca0282 ! -o br-121978ca0282 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -i br-00db5844bbb0 ! -o br-00db5844bbb0 -j DOCKER-ISOLATION-STAGE-2\n-A DOCKER-ISOLATION-STAGE-1 -j RETURN\n#<\/pre>\n\n\n\n
So, restarted gnmi-gateway, still same issue. Ok, I decided to check if the packets were actually hitting r01.<\/p>\n\n\n\n
So at first sight, the tcp handshake is established but then there is TCP RST….<\/p>\n\n\n\n
So I double checked that gnmi was runnig in my side:<\/p>\n\n\n\n
r1#show management api gnmi \nEnabled:            Yes\nServer:             running on port 3333, in MGMT VRF\nSSL Profile:        none\nQoS DSCP:           none\nr1#<\/code><\/pre>\n\n\n\n
At that moment, I thought that was an issue in cEOS… checking logs I couldnt see any confirmation but I decided to give it a go with vEOS that is more feature rich. So I turned up my GCP lab and followed the same steps  with gnmi-gateway. I updated the targets.json with the details of one of my vEOS devices. And run again:<\/p>\n\n\n\n
~\/gnmi\/gnmi-gateway release$ sudo docker run -it --rm -p 59100:59100 -v $(pwd)\/examples\/gnmi-prometheus\/targets.json:\/opt\/gnmi-gateway\/targets.json --name gnmi-gateway-01 --network gnmi-net gnmi-gateway:latest\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Starting GNMI Gateway.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Clustering is NOT enabled. No locking or cluster coordination will happen.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Starting connection manager.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Starting gNMI server on 0.0.0.0:9339.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Starting Prometheus exporter.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Connection manager received a target control message: 1 inserts 0 removes\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Initializing target gcp-r1 ([192.168.249.4:3333]) map[NoTLS:yes].\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Target gcp-r1: Connecting\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Target gcp-r1: Subscribing\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:20Z\",\"message\":\"Starting Prometheus HTTP server.\"}\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:30Z\",\"message\":\"Target gcp-r1: Disconnected\"}\nE1107 19:22:30.048410 1 reconnect.go:114] client.Subscribe (target \"gcp-r1\") failed: client \"gnmi\" : client \"gnmi\" : Dialer(192.168.249.4:3333, 10s): context deadline exceeded; reconnecting in 552.330144ms\n{\"level\":\"info\",\"time\":\"2020-11-07T19:22:40Z\",\"message\":\"Target gcp-r1: Disconnected\"}\nE1107 19:22:40.603141 1 reconnect.go:114] client.Subscribe (target \"gcp-r1\") failed: client \"gnmi\" : client \"gnmi\" : Dialer(192.168.249.4:3333, 10s): context deadline exceeded; reconnecting in 1.080381816s<\/pre>\n\n\n\n
Again, same issue. Let’s see from vEOS perspective.<\/p>\n\n\n\n
bash-4.2# tcpdump -i any tcp port 3333 -nnn\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode\nlistening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes\n18:52:31.874137 In 1e:3d:5b:13:d8:fe ethertype IPv4 (0x0800), length 76: 10.128.0.4.56546 > 192.168.249.4.3333: Flags [S], seq 4076065498, win 64240, options [mss 1460,sackOK,TS val 1752943121 ecr 0,nop,wscale 7], length 0\n18:52:31.874579 Out 50:00:00:04:00:00 ethertype IPv4 (0x0800), length 76: 192.168.249.4.3333 > 10.128.0.4.56546: Flags [S.], seq 3922060793, ack 4076065499, win 28960, options [mss 1460,sackOK,TS val 433503 ecr 1752943121,nop,wscale 7], length 0\n18:52:31.875882 In 1e:3d:5b:13:d8:fe ethertype IPv4 (0x0800), length 68: 10.128.0.4.56546 > 192.168.249.4.3333: Flags [.], ack 1, win 502, options [nop,nop,TS val 1752943123 ecr 433503], length 0\n18:52:31.876284 In 1e:3d:5b:13:d8:fe ethertype IPv4 (0x0800), length 320: 10.128.0.4.56546 > 192.168.249.4.3333: Flags [P.], seq 1:253, ack 1, win 502, options [nop,nop,TS val 1752943124 ecr 433503], length 252\n18:52:31.876379 Out 50:00:00:04:00:00 ethertype IPv4 (0x0800), length 68: 192.168.249.4.3333 > 10.128.0.4.56546: Flags [.], ack 253, win 235, options [nop,nop,TS val 433504 ecr 1752943124], length 0\n18:52:31.929448 Out 50:00:00:04:00:00 ethertype IPv4 (0x0800), length 89: 192.168.249.4.3333 > 10.128.0.4.56546: Flags [P.], seq 1:22, ack 253, win 235, options [nop,nop,TS val 433517 ecr 1752943124], length 21\n18:52:31.930028 In 1e:3d:5b:13:d8:fe ethertype IPv4 (0x0800), length 68: 10.128.0.4.56546 > 192.168.249.4.3333: Flags [.], ack 22, win 502, options [nop,nop,TS val 1752943178 ecr 433517], length 0\n18:52:31.930090 In 1e:3d:5b:13:d8:fe ethertype IPv4 (0x0800), length 68: 10.128.0.4.56546 > 192.168.249.4.3333: Flags [F.], seq 253, ack 22, win 502, options [nop,nop,TS val 1752943178 ecr 433517], length 0\n18:52:31.931603 Out 50:00:00:04:00:00 ethertype IPv4 (0x0800), length 68: 192.168.249.4.3333 > 10.128.0.4.56546: Flags [R.], seq 22, ack 254, win 235, options [nop,nop,TS val 433517 ecr 1752943178], length 0<\/pre>\n\n\n\n
So again in GCP, tcp is established but then TCP RST. As vEOS is my last resort, I tried to dig into that TCP connection. I downloaded a pcap to analyze with wireshark so get a better visual clue…<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
So, somehow, gnmi-gateway is trying to negotiate TLS!!! As per my understanding, my targets.json was configured with “NoTLS”: “yes” so that should be avoid, shouldn’t be?<\/p>\n\n\n\n
At that moment, I wanted to know how to identfiy TLS\/SSL packets using tcpdump as it is not always that easy to get quickly a pcap in wireshark. So I found the answer here<\/a>:<\/p>\n\n\n\n
bash-4.2# tcpdump -i any \"tcp port 3333 and (tcp[((tcp[12] & 0xf0) >> 2)] = 0x16)\"\ntcpdump: verbose output suppressed, use -v or -vv for full protocol decode\nlistening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes\n19:47:01.367197 In 1e:3d:5b:13:d8:fe (oui Unknown) ethertype IPv4 (0x0800), length 320: 10.128.0.4.50486 > 192.168.249.4.dec-notes: Flags [P.], seq 2715923852:2715924104, ack 2576249027, win 511, options [nop,nop,TS val 1194424180 ecr 1250876], length 252\n19:47:02.405870 In 1e:3d:5b:13:d8:fe (oui Unknown) ethertype IPv4 (0x0800), length 320: 10.128.0.4.50488 > 192.168.249.4.dec-notes: Flags [P.], seq 680803294:680803546, ack 3839769659, win 511, options [nop,nop,TS val 1194425218 ecr 1251136], length 252\n19:47:04.139458 In 1e:3d:5b:13:d8:fe (oui Unknown) ethertype IPv4 (0x0800), length 320: 10.128.0.4.50490 > 192.168.249.4.dec-notes: Flags [P.], seq 3963338234:3963338486, ack 1760248652, win 511, options [nop,nop,TS val 1194426952 ecr 1251569], length 252<\/pre>\n\n\n\n
Not something easy to remember \ud83d\ude41<\/p>\n\n\n\n
Ok, I wanted to be sure that gnmi was functional in vEOS and by a quick internet look up, I found this project gnmic<\/a>! Great job by the author!<\/p>\n\n\n\n
So I configured the tool and tested with my vEOS. And worked (without needing TLS)<\/p>\n\n\n\n
~\/gnmi\/gnmi-gateway release$ gnmic -a 192.168.249.4:3333 -u xxx -p xxx --insecure --insecure get \\\n--path \"\/interfaces\/interface[name=*]\/subinterfaces\/subinterface[index=*]\/ipv4\/addresses\/address\/config\/ip\"<\/code>\nGet Response:\n[\n{\n\"time\": \"1970-01-01T00:00:00Z\",\n\"updates\": [\n{\n\"Path\": \"interfaces\/interface[name=Management1]\/subinterfaces\/subinterface[index=0]\/ipv4\/addresses\/address[ip=192.168.249.4]\/config\/ip\",\n\"values\": {\n\"interfaces\/interface\/subinterfaces\/subinterface\/ipv4\/addresses\/address\/config\/ip\": \"192.168.249.4\"\n}\n},\n{\n\"Path\": \"interfaces\/interface[name=Ethernet2]\/subinterfaces\/subinterface[index=0]\/ipv4\/addresses\/address[ip=10.0.13.1]\/config\/ip\",\n\"values\": {\n\"interfaces\/interface\/subinterfaces\/subinterface\/ipv4\/addresses\/address\/config\/ip\": \"10.0.13.1\"\n}\n},\n{\n\"Path\": \"interfaces\/interface[name=Ethernet3]\/subinterfaces\/subinterface[index=0]\/ipv4\/addresses\/address[ip=192.168.1.1]\/config\/ip\",\n\"values\": {\n\"interfaces\/interface\/subinterfaces\/subinterface\/ipv4\/addresses\/address\/config\/ip\": \"192.168.1.1\"\n}\n},\n{\n\"Path\": \"interfaces\/interface[name=Ethernet1]\/subinterfaces\/subinterface[index=0]\/ipv4\/addresses\/address[ip=10.0.12.1]\/config\/ip\",\n\"values\": {\n\"interfaces\/interface\/subinterfaces\/subinterface\/ipv4\/addresses\/address\/config\/ip\": \"10.0.12.1\"\n}\n},\n{\n\"Path\": \"interfaces\/interface[name=Loopback1]\/subinterfaces\/subinterface[index=0]\/ipv4\/addresses\/address[ip=10.0.0.1]\/config\/ip\",\n\"values\": {\n\"interfaces\/interface\/subinterfaces\/subinterface\/ipv4\/addresses\/address\/config\/ip\": \"10.0.0.1\"\n}\n},\n{\n\"Path\": \"interfaces\/interface[name=Loopback2]\/subinterfaces\/subinterface[index=0]\/ipv4\/addresses\/address[ip=192.168.0.1]\/config\/ip\",\n\"values\": {\n\"interfaces\/interface\/subinterfaces\/subinterface\/ipv4\/addresses\/address\/config\/ip\": \"192.168.0.1\"\n}\n}\n]\n}\n]\n~\/gnmi\/gnmi-gateway release$<\/pre>\n\n\n\n
So, I kind of I was sure that my issue was configuring gnmi-gateway. I tried to troubleshoot it: removed the NoTLS, using the debugging mode, build the code,  read the Go code for Target (too complex for my Goland knowledge \ud83d\ude41 )<\/p>\n\n\n\n
So at the end, I gave up and opened an issue<\/a> with gnmi-gateway author. And he answered super quick with the solution!!! I misunderstood the meaning of “NoTLS” \ud83d\ude41<\/p>\n\n\n\n
So I followed his instructions to configure TLS in my gnmi cEOS config <\/p>\n\n\n\n
security pki certificate generate self-signed r01.crt key r01.key generate rsa 2048 validity 30000 parameters common-name r01\n!\nmanagement api gnmi\ntransport grpc GRPC\nssl profile SELFSIGNED\nport 3333\n!\n...\n!\nmanagement security\nssl profile SELFSIGNED\ncertificate r01.crt key r01.key\n!\nend<\/pre>\n\n\n\n
and all worked straightaway!<\/p>\n\n\n\n
~\/storage\/technology\/gnmi-gateway release$ docker run -it --rm -p 59100:59100 -v $(pwd)\/examples\/gnmi-prometheus\/targets.json:\/opt\/gnmi-gateway\/targets.json --name gnmi-gateway-01 --network gnmi-net gnmi-gateway:latest\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Starting GNMI Gateway.\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Clustering is NOT enabled. No locking or cluster coordination will happen.\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Starting connection manager.\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Starting gNMI server on 0.0.0.0:9339.\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Starting Prometheus exporter.\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Connection manager received a target control message: 1 inserts 0 removes\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Initializing target r01 ([172.20.0.2:3333]) map[NoTLS:yes].\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Target r01: Connecting\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Target r01: Subscribing\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Target r01: Connected\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:15Z\",\"message\":\"Target r01: Synced\"}<\/strong>\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:16Z\",\"message\":\"Starting Prometheus HTTP server.\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:39:45Z\",\"message\":\"Connection manager received a target control message: 1 inserts 0 removes\"}\n{\"level\":\"info\",\"time\":\"2020-11-08T09:40:15Z\",\"message\":\"Connection manager received a target control message: 1 inserts 0 removes\"}<\/pre>\n\n\n\n
So I can start prometheus<\/p>\n\n\n\n
~\/storage\/technology\/gnmi-gateway release$ docker run \\\n-it --rm \\\n-p 9090:9090 \\\n-v $(pwd)\/examples\/gnmi-prometheus\/prometheus.yml:\/etc\/prometheus\/prometheus.yml \\\n--name prometheus-01 \\\n--network gnmi-net \\\nprom\/prometheus\nUnable to find image 'prom\/prometheus:latest' locally\nlatest: Pulling from prom\/prometheus\n76df9210b28c: Pull complete\n559be8e06c14: Pull complete\n66945137dd82: Pull complete\n8cbce0960be4: Pull complete\nf7bd1c252a58: Pull complete\n6ad12224c517: Pull complete\nee9cd36fa25a: Pull complete\nd73034c1b9c3: Pull complete\nb7103b774752: Pull complete\n2ba5d8ece07a: Pull complete\nab11729a0297: Pull complete\n1549b85a3587: Pull complete\nDigest: sha256:b899dbd1b9017b9a379f76ce5b40eead01a62762c4f2057eacef945c3c22d210\nStatus: Downloaded newer image for prom\/prometheus:latest\nlevel=info ts=2020-11-08T09:40:26.622Z caller=main.go:315 msg=\"No time or size retention was set so using the default time retention\" duration=15d\nlevel=info ts=2020-11-08T09:40:26.623Z caller=main.go:353 msg=\"Starting Prometheus\" version=\"(version=2.22.1, branch=HEAD, revision=00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)\"\nlevel=info ts=2020-11-08T09:40:26.623Z caller=main.go:358 build_context=\"(go=go1.15.3, user=root@516b109b1732, date=20201105-14:02:25)\"\nlevel=info ts=2020-11-08T09:40:26.623Z caller=main.go:359 host_details=\"(Linux 5.9.0-1-amd64 #1 SMP Debian 5.9.1-1 (2020-10-17) x86_64 b0fadf4a4c80 (none))\"\nlevel=info ts=2020-11-08T09:40:26.623Z caller=main.go:360 fd_limits=\"(soft=1048576, hard=1048576)\"\nlevel=info ts=2020-11-08T09:40:26.623Z caller=main.go:361 vm_limits=\"(soft=unlimited, hard=unlimited)\"\nlevel=info ts=2020-11-08T09:40:26.641Z caller=main.go:712 msg=\"Starting TSDB \u2026\"\nlevel=info ts=2020-11-08T09:40:26.641Z caller=web.go:516 component=web msg=\"Start listening for connections\" address=0.0.0.0:9090\nlevel=info ts=2020-11-08T09:40:26.668Z caller=head.go:642 component=tsdb msg=\"Replaying on-disk memory mappable chunks if any\"\nlevel=info ts=2020-11-08T09:40:26.669Z caller=head.go:656 component=tsdb msg=\"On-disk memory mappable chunks replay completed\" duration=103.51\u00b5s\nlevel=info ts=2020-11-08T09:40:26.669Z caller=head.go:662 component=tsdb msg=\"Replaying WAL, this may take a while\"\nlevel=info ts=2020-11-08T09:40:26.672Z caller=head.go:714 component=tsdb msg=\"WAL segment loaded\" segment=0 maxSegment=0\nlevel=info ts=2020-11-08T09:40:26.672Z caller=head.go:719 component=tsdb msg=\"WAL replay completed\" checkpoint_replay_duration=123.684\u00b5s wal_replay_duration=2.164743ms total_replay_duration=3.357021ms\nlevel=info ts=2020-11-08T09:40:26.675Z caller=main.go:732 fs_type=2fc12fc1\nlevel=info ts=2020-11-08T09:40:26.676Z caller=main.go:735 msg=\"TSDB started\"\nlevel=info ts=2020-11-08T09:40:26.676Z caller=main.go:861 msg=\"Loading configuration file\" filename=\/etc\/prometheus\/prometheus.yml\nlevel=info ts=2020-11-08T09:40:26.684Z caller=main.go:892 msg=\"Completed loading of configuration file\" filename=\/etc\/prometheus\/prometheus.yml totalDuration=7.601103ms remote_storage=22.929\u00b5s web_handler=623ns query_engine=1.64\u00b5s scrape=5.517391ms scrape_sd=359.447\u00b5s notify=18.349\u00b5s notify_sd=3.921\u00b5s rules=15.744\u00b5s\nlevel=info ts=2020-11-08T09:40:26.685Z caller=main.go:684 msg=\"Server is ready to receive web requests.\"<\/pre>\n\n\n\n
Now we can open prometheus UI and verify if we are consuming data from cEOS r01.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Yeah! it is there.<\/p>\n\n\n\n
So all working at then. It has a nice experience. At the end of the day, I want to know more about gNMI\/protobuffer, etc. The cold thing here is you can get telemetry and configuration management of your devices. So using gnmi-gateway (that is more for a high availability env like Netflix) and gnmic are great tools to get your head around.<\/p>\n\n\n\n
Other lab I want to try is this eos-gnmi-telemetry-grafana<\/a>.<\/p>\n\n\n\n
The to-do list always keeps growing.<\/p>\n","protected":false},"excerpt":{"rendered":"
I was already playing with gNMI and protobuf a couple of months ago. But this week I received a summary from the last NANOG80 meeting and there was a presentation about it. Great job from Colin! So I decided to give it a go as the demo was based on docker and I have already … Continue reading “gnmi-ssl-p2”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,22,2],"tags":[],"class_list":["post-469","post","type-post","status-publish","format-standard","hentry","category-automation","category-monitoring","category-networks"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/469","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=469"}],"version-history":[{"count":2,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/469\/revisions"}],"predecessor-version":[{"id":1327,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/469\/revisions\/1327"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=469"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=469"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=469"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}