As usual, I am following Anton’s blog and now I want to follow his series about Protobuf/gNMI. All merit and hard work is for the author. I am just doing copy/paste. All his code related to this topic is in his github repo:
First time I heard about protobuf was in the context of telemetry from Arista LANZ (44.3.7)
Now it is my chance to get some knowledge about it. Protobuf is a new data encoding type (like JSON) meant for speed mainly. Mayor things, this is a binary protocol. And we are going to use Protobuf to encode YANG/OpenConfig. And the transport protocol is going to be gNMI.
Index
- 0- Create python env
- 1- Install protobuf
- 2- Create and compile protobuf file for the OpenConfig modules openconfig-interfaces.yang.
- 3- Create python script to write protobuf message based on the model compiled earlier
- 4- Create python script to read that protobuf message
- 5- Use gNMI: Create python script to get interface configuration from cEOS
0- Create Python Env
$ mkdir protobuf $ cd protobuf $ python3 -m virtualenv ENV $ source ENV/bin/activate $ python -m pip install grpcio $ python -m pip install grpcio-tools $ python -m pip install pyang
1- Install protobuf
For debian:
$ sudo aptitude install protobuf-compile $ protoc --version libprotoc 3.12.3
2- Create and compile protobuf file
This is a quite difficult part. Try to install “pyang” for python and clone openconfig. Keep in mind that I have removed “ro” entries manually below:
$ ls -ltr total 11 -rw-r--r-- 1 tomas tomas 1240 Aug 19 18:37 README.md -rw-r--r-- 1 tomas tomas 11358 Aug 19 18:37 LICENSE drwxr-xr-x 3 tomas tomas 4 Aug 19 18:37 release drwxr-xr-x 4 tomas tomas 12 Aug 19 18:37 doc drwxr-xr-x 3 tomas tomas 4 Aug 19 18:37 third_party $ $ pyang -f tree -p ./release/models/ ./release/models/interfaces/openconfig-interfaces.yang module: openconfig-interfaces +--rw interfaces +--rw interface* [name] +--rw name -> ../config/name +--rw config | +--rw name? string | +--rw type identityref | +--rw mtu? uint16 | +--rw loopback-mode? boolean | +--rw description? string | +--rw enabled? boolean +--rw hold-time | +--rw config | | +--rw up? uint32 | | +--rw down? uint32 +--rw subinterfaces +--rw subinterface* [index] +--rw index -> ../config/index +--rw config +--rw index? uint32 +--rw description? string +--rw enabled? boolean
So this is the YANG model that we want to transform into protobuf.
To be honest, If I have to match that output with the content of the file itself, I dont understant it.
As Anton mentions, you need to check the official protobuf guide and protobuf python guide to create the proto file for the interface YANG model. These two links explain the structure of our new protofile.
In one side, I think I understand the process of converting YANG to Protobug. But I should try something myself to be sure 🙂
The .proto code doesn’t appear properly formatted in my blog so you can see it in the fig above or in github.
Compile:
$ protoc -I=. --python_out=. openconfig_interfaces.proto $ ls -ltr | grep openconfig_interfaces -rw-r--r-- 1 tomas tomas 1247 Aug 20 14:01 openconfig_interfaces.proto -rw-r--r-- 1 tomas tomas 20935 Aug 20 14:03 openconfig_interfaces_pb2.py
3- Create python script to write protobuf
The script has a dict “intend” to be used to populate the proto message. Once it is populated with the info, it is written to a file as byte stream.
$ python create_protobuf.py oc_if.bin $ file oc_if.bin oc_if.bin: data
4- Create python script to read protobuf
This is based on the next blog entry of Anton’s series.
The script that read the protobuf message is here.
$ python read_protobuf.py oc_if.bin
{'interfaces': {'interface': [{'name': 'Ethernet1', 'config': {'name': 'Ethernet1', 'type': 0, 'mtu': 1514, 'description': 'ABC', 'enabled': True, 'subinterfaces': {'subinterface': [{'index': 0, 'config': {'index': 0, 'description': 'DEF', 'enabled': True}}]}}}, {'name': 'Ethernet2', 'config': {'name': 'Ethernet2', 'type': 0, 'mtu': 1514, 'description': '123', 'enabled': True, 'subinterfaces': {'subinterface': [{'index': 0, 'config': {'index': 0, 'description': '456', 'enabled': True}}]}}}]}}
$
5- Use gNMI with cEOS
This part is based in the third blog from Anton.
The challenge here is how he found out what files to use.
$ ls -ltr gnmi/proto/gnmi/ total 62 -rw-r--r-- 1 tomas tomas 21907 Aug 20 15:10 gnmi.proto -rw-r--r-- 1 tomas tomas 125222 Aug 20 15:10 gnmi.pb.go -rw-r--r-- 1 tomas tomas 76293 Aug 20 15:10 gnmi_pb2.py -rw-r--r-- 1 tomas tomas 4864 Aug 20 15:10 gnmi_pb2_grpc.py $ $ ls -ltr gnmi/proto/gnmi_ext/ total 14 -rw-r--r-- 1 tomas tomas 2690 Aug 20 15:10 gnmi_ext.proto -rw-r--r-- 1 tomas tomas 19013 Aug 20 15:10 gnmi_ext.pb.go -rw-r--r-- 1 tomas tomas 10191 Aug 20 15:10 gnmi_ext_pb2.py -rw-r--r-- 1 tomas tomas 83 Aug 20 15:10 gnmi_ext_pb2_grpc.py $
I can see the blog and github doesnt match and I can’t really follow. Based on that, I have created an script to get the interface config from one cEOS switch using gNMI interface:
$ cat gnmi_get_if_config.py
#!/usr/bin/env python
# Modules
import grpc
from bin.gnmi_pb2_grpc import *
from bin.gnmi_pb2 import *
import json
import pprint
# Own modules
from bin.PathGenerator import gnmi_path_generator
# Variables
path = {'inventory': 'inventory.json'}
info_to_collect = ['openconfig-interfaces:interfaces']
# User-defined functions
def json_to_dict(path):
with open(path, 'r') as f:
return json.loads(f.read())
# Body
if __name__ == '__main__':
inventory = json_to_dict(path['inventory'])
for td_entry in inventory['devices']:
metadata = [('username', td_entry['username']), ('password', td_entry['password'])]
channel = grpc.insecure_channel(f'{td_entry["ip_address"]}:{td_entry["port"]}', metadata)
grpc.channel_ready_future(channel).result(timeout=5)
stub = gNMIStub(channel)
for itc_entry in info_to_collect:
print(f'Getting data for {itc_entry} from {td_entry["hostname"]} over gNMI...\n')
intent_path = gnmi_path_generator(itc_entry)
print("gnmi_path:\n")
print(intent_path)
gnmi_message_request = GetRequest(path=[intent_path], type=0, encoding=4)
gnmi_message_response = stub.Get(gnmi_message_request, metadata=metadata)
# we get the outout of gnmi_response that is json as string of bytes
x = gnmi_message_response.notification[0].update[0].val.json_ietf_val
# decode the string of bytes as string and then transform to pure json
y = json.loads(x.decode('utf-8'))
#import ipdb; ipdb.set_trace()
# print nicely json
pprint.pprint(y)
This is my cEOS config:
r01#show management api gnmi Enabled: Yes Server: running on port 3333, in default VRF SSL Profile: none QoS DSCP: none r01# r01# r01#show version cEOSLab Hardware version: Serial number: Hardware MAC address: 0242.ac8d.adef System MAC address: 0242.ac8d.adef Software image version: 4.23.3M Architecture: i686 Internal build version: 4.23.3M-16431779.4233M Internal build ID: afb8ec89-73bd-4410-b090-f000f70505bb cEOS tools version: 1.1 Uptime: 6 weeks, 1 days, 3 hours and 13 minutes Total memory: 8124244 kB Free memory: 1923748 kB r01# r01# r01#show ip interface brief Address Interface IP Address Status Protocol MTU Owner Ethernet1 10.0.12.1/30 up up 1500 Ethernet2 10.0.13.1/30 up up 1500 Loopback1 10.0.0.1/32 up up 65535 Loopback2 192.168.0.1/32 up up 65535 Vlan100 1.1.1.1/24 up up 1500 r01#
And it seems to work:
$ python gnmi_get_if_config.py
Getting data for openconfig-interfaces:interfaces from r01 over gNMI…
gnmi_path:
origin: "openconfig-interfaces"
elem {
name: "interfaces"
}
{'openconfig-interfaces:interface': [{'config': {'arista-intf-augments:load-interval': 300,
'description': '',
'enabled': True,
'loopback-mode': False,
'mtu': 0,
'name': 'Ethernet2',
'openconfig-vlan:tpid': 'openconfig-vlan-types:TPID_0X8100',
'type': 'iana-if-type:ethernetCsmacd'},
Summary
It has been be interesting to play with ProtoBug and gNMI but I have just grasped the surface.
Notes
My test env is here.
Other Info
ripe78 cisco telemetry.
cisco live 2019 intro to gRPC
gRPC and GPB for network engineers here.