{"id":380,"date":"2020-08-20T15:31:43","date_gmt":"2020-08-20T14:31:43","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=380"},"modified":"2020-08-20T17:35:57","modified_gmt":"2020-08-20T16:35:57","slug":"protobuf-gnmi","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2020\/08\/20\/protobuf-gnmi\/","title":{"rendered":"Protobuf\/gNMI"},"content":{"rendered":"\n<p>As usual, I am following <a href=\"https:\/\/karneliuk.com\/\">Anton&#8217;s blog<\/a> and now I want to follow his series about <a href=\"https:\/\/karneliuk.com\/2020\/05\/gnmi-part-1-intro-to-protobuf\/\">Protobuf\/gNMI<\/a>.  All merit and hard work is for the author. I am just doing copy\/paste. All his code related to this topic is in his github <a href=\"https:\/\/github.com\/akarneliuk\/grpc_demo\">repo<\/a>:<\/p>\n\n\n\n<p>First time I heard about protobuf was in the context of telemetry from <a href=\"https:\/\/www.arista.com\/assets\/data\/pdf\/user-manual\/um-eos\/Chapters\/Latency%20Analyzer%20(LANZ).pdf\">Arista LANZ<\/a> (44.3.7)<\/p>\n\n\n\n<p>Now it is my chance to get some knowledge about it. <a href=\"https:\/\/developers.google.com\/protocol-buffers\">Protobuf<\/a> is a new data encoding type (like JSON) meant for speed mainly. Mayor things, this is a binary protocol. And we are going to use Protobuf to encode YANG\/<a href=\"https:\/\/github.com\/openconfig\/public\">OpenConfig<\/a>. And the transport protocol is going to be gNMI.<\/p>\n\n\n\n<p>Index<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>0- Create python env<\/li><li>1- Install protobuf<\/li><li>2- Create and compile protobuf file for the OpenConfig modules openconfig-interfaces.yang.<\/li><li>3- Create python script to write protobuf message based on the model compiled earlier<\/li><li>4- Create python script to read that protobuf message<\/li><li>5- Use gNMI: Create python script to get interface configuration from cEOS<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">0- Create Python Env<\/h2>\n\n\n\n<pre class=\"wp-block-preformatted\">$ mkdir protobuf\n$ cd protobuf\n$ python3 -m virtualenv ENV\n$ source ENV\/bin\/activate\n$ python -m pip install grpcio\n$ python -m pip install grpcio-tools\n$ python -m pip install pyang<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">1- Install protobuf<\/h2>\n\n\n\n<p>For debian:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ sudo aptitude install protobuf-compile\n$ protoc --version\nlibprotoc 3.12.3<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">2- Create and compile protobuf file<\/h2>\n\n\n\n<p>This is a quite difficult part. Try to install &#8220;pyang&#8221; for python and clone <a href=\"https:\/\/github.com\/openconfig\/public\">openconfig<\/a>. Keep in mind that I have removed &#8220;ro&#8221; entries manually below:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ ls -ltr\ntotal 11\n-rw-r--r-- 1 tomas tomas 1240 Aug 19 18:37 README.md\n-rw-r--r-- 1 tomas tomas 11358 Aug 19 18:37 LICENSE\ndrwxr-xr-x 3 tomas tomas 4 Aug 19 18:37 release\ndrwxr-xr-x 4 tomas tomas 12 Aug 19 18:37 doc\ndrwxr-xr-x 3 tomas tomas 4 Aug 19 18:37 third_party\n$\n$ pyang -f tree -p .\/release\/models\/ .\/release\/models\/interfaces\/openconfig-interfaces.yang\nmodule: openconfig-interfaces\n+--rw interfaces\n+--rw interface* [name]\n+--rw name -&gt; ..\/config\/name\n+--rw config\n| +--rw name? string\n| +--rw type identityref\n| +--rw mtu? uint16\n| +--rw loopback-mode? boolean\n| +--rw description? string\n| +--rw enabled? boolean\n+--rw hold-time\n| +--rw config\n| | +--rw up? uint32\n| | +--rw down? uint32\n+--rw subinterfaces\n+--rw subinterface* [index]\n+--rw index -&gt; ..\/config\/index\n+--rw config\n  +--rw index? uint32\n  +--rw description? string\n  +--rw enabled? boolean\n<\/pre>\n\n\n\n<p>So this is the YANG model that we want to transform into protobuf.<\/p>\n\n\n\n<p>To be honest, If I have to match that output with the content of the file itself, I dont understant it.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"440\" src=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-1024x440.png\" alt=\"\" class=\"wp-image-382\" srcset=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-1024x440.png 1024w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-300x129.png 300w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-768x330.png 768w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-1536x660.png 1536w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-2048x880.png 2048w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-15-36-1-1200x516.png 1200w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/figure>\n\n\n\n<p>As Anton mentions, you need to check the <a href=\"https:\/\/developers.google.com\/protocol-buffers\/docs\/proto3\">official protobuf guide<\/a> and <a href=\"https:\/\/developers.google.com\/protocol-buffers\/docs\/pythontutorial\">protobuf python guide<\/a> to create the proto file for the interface YANG model. These two links explain the structure of our new protofile.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"631\" src=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-1024x631.png\" alt=\"\" class=\"wp-image-383\" srcset=\"https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-1024x631.png 1024w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-300x185.png 300w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-768x473.png 768w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-1536x946.png 1536w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-2048x1262.png 2048w, https:\/\/blog.thomarite.uk\/wp-content\/uploads\/2020\/08\/Screenshot-from-2020-08-20-10-25-11-1200x739.png 1200w\" sizes=\"auto, (max-width: 709px) 85vw, (max-width: 909px) 67vw, (max-width: 1362px) 62vw, 840px\" \/><\/figure>\n\n\n\n<p>In one side, I think I understand the process of converting YANG to Protobug. But I should try something myself to be sure \ud83d\ude42<\/p>\n\n\n\n<p>The .proto code doesn&#8217;t appear properly formatted in my blog so you can see it in the fig above or in github.<\/p>\n\n\n\n<p>Compile:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ protoc -I=. --python_out=. openconfig_interfaces.proto\n$ ls -ltr | grep openconfig_interfaces\n-rw-r--r-- 1 tomas tomas 1247 Aug 20 14:01 openconfig_interfaces.proto\n-rw-r--r-- 1 tomas tomas 20935 Aug 20 14:03 openconfig_interfaces_pb2.py<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">3- Create python script to write protobuf<\/h2>\n\n\n\n<p>The <a href=\"https:\/\/github.com\/akarneliuk\/grpc_demo\/blob\/master\/create_protobuf.py\">script<\/a> has a dict &#8220;intend&#8221; to be used to populate the proto message. Once it is populated with the info, it is written to a file as byte stream.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ python create_protobuf.py oc_if.bin\n$ file oc_if.bin\noc_if.bin: data<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">4- Create python script to read protobuf<\/h2>\n\n\n\n<p>This is based on the next <a href=\"https:\/\/karneliuk.com\/2020\/05\/gnmi-part-2-decoding-protobuf-messages-with-python\/\">blog<\/a> entry of Anton&#8217;s series.<\/p>\n\n\n\n<p>The script that read the protobuf message is <a href=\"https:\/\/github.com\/akarneliuk\/grpc_demo\/blob\/master\/read_protobuf.py\">here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ python read_protobuf.py oc_if.bin\n{'interfaces': {'interface': [{'name': 'Ethernet1', 'config': {'name': 'Ethernet1', 'type': 0, 'mtu': 1514, 'description': 'ABC', 'enabled': True, 'subinterfaces': {'subinterface': [{'index': 0, 'config': {'index': 0, 'description': 'DEF', 'enabled': True}}]}}}, {'name': 'Ethernet2', 'config': {'name': 'Ethernet2', 'type': 0, 'mtu': 1514, 'description': '123', 'enabled': True, 'subinterfaces': {'subinterface': [{'index': 0, 'config': {'index': 0, 'description': '456', 'enabled': True}}]}}}]}}\n$<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">5- Use gNMI with cEOS<\/h2>\n\n\n\n<p>This part is based in the third <a href=\"https:\/\/karneliuk.com\/2020\/05\/gnmi-part-3-using-grpc-to-collect-data-in-openconfig-yang-from-arista-eos-and-nokia-sr-os\/\">blog<\/a> from Anton.<\/p>\n\n\n\n<p>The challenge here is how he found out what files to use.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ ls -ltr gnmi\/proto\/gnmi\/\ntotal 62\n-rw-r--r-- 1 tomas tomas 21907 Aug 20 15:10 gnmi.proto\n-rw-r--r-- 1 tomas tomas 125222 Aug 20 15:10 gnmi.pb.go\n-rw-r--r-- 1 tomas tomas 76293 Aug 20 15:10 gnmi_pb2.py\n-rw-r--r-- 1 tomas tomas 4864 Aug 20 15:10 gnmi_pb2_grpc.py\n$\n$ ls -ltr gnmi\/proto\/gnmi_ext\/\ntotal 14\n-rw-r--r-- 1 tomas tomas 2690 Aug 20 15:10 gnmi_ext.proto\n-rw-r--r-- 1 tomas tomas 19013 Aug 20 15:10 gnmi_ext.pb.go\n-rw-r--r-- 1 tomas tomas 10191 Aug 20 15:10 gnmi_ext_pb2.py\n-rw-r--r-- 1 tomas tomas 83 Aug 20 15:10 gnmi_ext_pb2_grpc.py\n$<\/pre>\n\n\n\n<p>I can see the blog and github doesnt match and  I can&#8217;t really follow. Based on that, I have created an script to get the interface config from one cEOS switch using gNMI interface:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>$ cat gnmi_get_if_config.py \n#!\/usr\/bin\/env python\n\n# Modules\nimport grpc\nfrom bin.gnmi_pb2_grpc import *\nfrom bin.gnmi_pb2 import *\nimport json\nimport pprint\n\n# Own modules\nfrom bin.PathGenerator import gnmi_path_generator\n\n# Variables\npath = {'inventory': 'inventory.json'}\ninfo_to_collect = &#91;'openconfig-interfaces:interfaces']\n\n\n# User-defined functions\ndef json_to_dict(path):\n    with open(path, 'r') as f:\n        return json.loads(f.read())\n\n\n# Body\nif __name__ == '__main__':\n    inventory = json_to_dict(path&#91;'inventory'])\n\n    for td_entry in inventory&#91;'devices']:\n        metadata = &#91;('username', td_entry&#91;'username']), ('password', td_entry&#91;'password'])]\n\n        channel = grpc.insecure_channel(f'{td_entry&#91;\"ip_address\"]}:{td_entry&#91;\"port\"]}', metadata)\n        grpc.channel_ready_future(channel).result(timeout=5)\n\n        stub = gNMIStub(channel)\n\n        for itc_entry in info_to_collect:\n            print(f'Getting data for {itc_entry} from {td_entry&#91;\"hostname\"]} over gNMI...\\n')\n\n            intent_path = gnmi_path_generator(itc_entry)\n            print(\"gnmi_path:\\n\")\n            print(intent_path)\n            gnmi_message_request = GetRequest(path=&#91;intent_path], type=0, encoding=4)\n            gnmi_message_response = stub.Get(gnmi_message_request, metadata=metadata)\n            # we get the outout of gnmi_response that is json as string of bytes\n            x = gnmi_message_response.notification&#91;0].update&#91;0].val.json_ietf_val\n            # decode the string of bytes as string and then transform to pure json\n            y = json.loads(x.decode('utf-8'))\n            #import ipdb; ipdb.set_trace()\n            # print nicely json\n            pprint.pprint(y)\n<\/code><\/pre>\n\n\n\n<p>This is my cEOS config:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">r01#show management api gnmi\nEnabled: Yes\nServer: running on port 3333, in default VRF\nSSL Profile: none\nQoS DSCP: none\nr01#\nr01#\nr01#show version\ncEOSLab\nHardware version:\nSerial number:\nHardware MAC address: 0242.ac8d.adef\nSystem MAC address: 0242.ac8d.adef\nSoftware image version: 4.23.3M\nArchitecture: i686\nInternal build version: 4.23.3M-16431779.4233M\nInternal build ID: afb8ec89-73bd-4410-b090-f000f70505bb\ncEOS tools version: 1.1\nUptime: 6 weeks, 1 days, 3 hours and 13 minutes\nTotal memory: 8124244 kB\nFree memory: 1923748 kB\nr01#\nr01#\nr01#show ip interface brief\nAddress\nInterface IP Address Status Protocol MTU Owner\n\nEthernet1 10.0.12.1\/30 up up 1500\nEthernet2 10.0.13.1\/30 up up 1500\nLoopback1 10.0.0.1\/32 up up 65535\nLoopback2 192.168.0.1\/32 up up 65535\nVlan100 1.1.1.1\/24 up up 1500\nr01#<\/pre>\n\n\n\n<p>And it seems to work:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ python gnmi_get_if_config.py\nGetting data for openconfig-interfaces:interfaces from r01 over gNMI\u2026\ngnmi_path:\norigin: \"openconfig-interfaces\"\nelem {\nname: \"interfaces\"\n}\n{'openconfig-interfaces:interface': [{'config': {'arista-intf-augments:load-interval': 300,\n'description': '',\n'enabled': True,\n'loopback-mode': False,\n'mtu': 0,\n'name': 'Ethernet2',\n'openconfig-vlan:tpid': 'openconfig-vlan-types:TPID_0X8100',\n'type': 'iana-if-type:ethernetCsmacd'},<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<p>It has been be interesting to play with ProtoBug and gNMI but I have just grasped the surface.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Notes<\/h2>\n\n\n\n<p>My test env is <a href=\"https:\/\/github.com\/thomarite\/protobug-gnmi\">here<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Other Info<\/h2>\n\n\n\n<p>ripe78 cisco <a href=\"https:\/\/ripe78.ripe.net\/presentations\/26-ripe78_Korshunov_Streaming_Telemetry_consideration_and_challenges_final.pdf\">telemetry<\/a>. <\/p>\n\n\n\n<p>cisco live 2019 intro to <a href=\"https:\/\/www.ciscolive.com\/c\/dam\/r\/ciscolive\/emea\/docs\/2019\/pdf\/DEVWKS-1381.pdf\">gRPC<\/a><\/p>\n\n\n\n<p>gRPC and GPB for network engineers <a href=\"https:\/\/github.com\/nleiva\/gmessaging\">here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As usual, I am following Anton&#8217;s blog and now I want to follow his series about Protobuf\/gNMI. All merit and hard work is for the author. I am just doing copy\/paste. All his code related to this topic is in his github repo: First time I heard about protobuf was in the context of telemetry &hellip; <a href=\"https:\/\/blog.thomarite.uk\/index.php\/2020\/08\/20\/protobuf-gnmi\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Protobuf\/gNMI&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,22],"tags":[],"class_list":["post-380","post","type-post","status-publish","format-standard","hentry","category-automation","category-monitoring"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/380","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=380"}],"version-history":[{"count":4,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/380\/revisions"}],"predecessor-version":[{"id":470,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/380\/revisions\/470"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=380"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=380"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=380"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}