{"id":270,"date":"2020-07-09T18:37:50","date_gmt":"2020-07-09T17:37:50","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=270"},"modified":"2020-07-21T17:39:50","modified_gmt":"2020-07-21T16:39:50","slug":"ansible-troubleshooting","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2020\/07\/09\/ansible-troubleshooting\/","title":{"rendered":"Ansible &#8211; Troubleshooting"},"content":{"rendered":"\n<p>A couple of years a go I wrote a playbook with ansible to use napalm for configuring some switches. <\/p>\n\n\n\n<p>I wanted to test again ansible as I am quite rusty and there is always demand for that.<\/p>\n\n\n\n<p>I started with just something basic with my ceos lab.<\/p>\n\n\n\n<p>All my code is here: <\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/thomarite\/ceos-testing\/tree\/master\/ansible\">https:\/\/github.com\/thomarite\/ceos-testing\/tree\/master\/ansible<\/a><\/p>\n\n\n\n<p>Initially I was following the official documentation for Arista EOS Ansible modules: <\/p>\n\n\n\n<p><a href=\"https:\/\/ansible-arista-howto.readthedocs.io\/en\/latest\/COLLECTING_STATUS.html\">https:\/\/ansible-arista-howto.readthedocs.io\/en\/latest\/COLLECTING_STATUS.html<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/titom73\/ansible-arista-module-howto\">https:\/\/github.com\/titom73\/ansible-arista-module-howto<\/a><\/p>\n\n\n\n<p>Installing ansible was fine with pip in my venv. <\/p>\n\n\n\n<p>But I hit the wall with just the first example using &#8220;eos_facts&#8221;. Initially I wasnt adding debugging flags so was even worse. Fortunately I remembered &#8220;-vvv&#8221;. I was seeing this:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">The full traceback is:\nTraceback (most recent call last):\nFile \"\/home\/tomas\/.ansible\/tmp\/ansible-tmp-1594296522.1539829-295453-189146847007138\/AnsiballZ_eos_facts.py\", line 102, in\n_ansiballz_main()\nFile \"\/home\/tomas\/.ansible\/tmp\/ansible-tmp-1594296522.1539829-295453-189146847007138\/AnsiballZ_eos_facts.py\", line 94, in _ansiballz_main\ninvoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\nFile \"\/home\/tomas\/.ansible\/tmp\/ansible-tmp-1594296522.1539829-295453-189146847007138\/AnsiballZ_eos_facts.py\", line 40, in invoke_module\nrunpy.run_module(mod_name='ansible.modules.network.eos.eos_facts', init_globals=None, run_name='<strong>main<\/strong>', alter_sys=True)\nFile \"\/home\/tomas\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/runpy.py\", line 205, in run_module\nreturn _run_module_code(code, init_globals, run_name, mod_spec)\nFile \"\/home\/tomas\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/runpy.py\", line 96, in _run_module_code\nmod_name, mod_spec, pkg_name, script_name)\nFile \"\/home\/tomas\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/runpy.py\", line 85, in _run_code\nexec(code, run_globals)\nFile \"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/modules\/network\/eos\/eos_facts.py\", line 206, in\nFile \"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/modules\/network\/eos\/eos_facts.py\", line 197, in main\nFile \"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/facts\/facts.py\", line 23, in <strong>init<\/strong>\nFile \"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/network.py\", line 213, in get_resource_connection\nFile \"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/network.py\", line 229, in get_capabilities\nFile \"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/connection.py\", line 121, in <strong>init<\/strong>\nAssertionError: socket_path must be a value\nfatal: [r3]: FAILED! => {\n\"changed\": false,\n\"module_stderr\": \"Traceback (most recent call last):\\n File \\\"\/home\/tomas\/.ansible\/tmp\/ansible-tmp-1594296522.1539829-295453-189146847007138\/AnsiballZ_eos_facts.py\\\", line 102, in \\n _ansiballz_main()\\n File \\\"\/home\/tomas\/.ansible\/tmp\/ansible-tmp-1594296522.1539829-295453-189146847007138\/AnsiballZ_eos_facts.py\\\", line 94, in _ansiballz_main\\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\\n File \\\"\/home\/tomas\/.ansible\/tmp\/ansible-tmp-1594296522.1539829-295453-189146847007138\/AnsiballZ_eos_facts.py\\\", line 40, in invoke_module\\n runpy.run_module(mod_name='ansible.modules.network.eos.eos_facts', init_globals=None, run_name='<strong>main<\/strong>', alter_sys=True)\\n File \\\"\/home\/tomas\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/runpy.py\\\", line 205, in run_module\\n return _run_module_code(code, init_globals, run_name, mod_spec)\\n File \\\"\/home\/tomas\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/runpy.py\\\", line 96, in _run_module_code\\n mod_name, mod_spec, pkg_name, script_name)\\n File \\\"\/home\/tomas\/.pyenv\/versions\/3.7.3\/lib\/python3.7\/runpy.py\\\", line 85, in _run_code\\n exec(code, run_globals)\\n File \\\"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/modules\/network\/eos\/eos_facts.py\\\", line 206, in \\n File \\\"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/modules\/network\/eos\/eos_facts.py\\\", line 197, in main\\n File \\\"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/facts\/facts.py\\\", line 23, in <strong>init<\/strong>\\n File \\\"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/network.py\\\", line 213, in get_resource_connection\\n File \\\"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/network.py\\\", line 229, in get_capabilities\\n File \\\"\/tmp\/ansible_eos_facts_payload_r5gz8rov\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/connection.py\\\", line 121, in <strong>init<\/strong>\\nAssertionError: socket_path must be a value\\n\",\n\"module_stdout\": \"\",\n\"msg\": \"MODULE FAILURE\\nSee stdout\/stderr for the exact error\",\n\"rc\": 1\n}<\/pre>\n\n\n\n<p>So, &#8220;socket_path&#8221; was defined. I checked all the python files mentioned in the stack but couldnt find anything.<\/p>\n\n\n\n<p>It was clear that I wasn&#8217;t providing enough info to ansible to establish the socket for connection to the devices (ip:port)<\/p>\n\n\n\n<p>And the example from the documentation didnt work neither:<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/network\/user_guide\/platform_eos.html#using-eapi-in-ansible\">https:\/\/docs.ansible.com\/ansible\/latest\/network\/user_guide\/platform_eos.html#using-eapi-in-ansible<\/a><\/p>\n\n\n\n<p>I knew my old ansible script was working before I left my job. But I knew as well that I was using the latest version of ansible so very likely things have changed since then.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ ansible --version\nansible 2.9.10<\/pre>\n\n\n\n<p>So I had to read about the &#8220;eos_fact&#8221; and &#8220;eos_config&#8221; module searching here:<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/modules\/list_of_network_modules.html#eos\">https:\/\/docs.ansible.com\/ansible\/latest\/modules\/list_of_network_modules.html<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/modules\/eos_facts_module.html#eos-facts-module\">https:\/\/docs.ansible.com\/ansible\/latest\/modules\/eos_facts_module.html#eos-facts-module<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/modules\/eos_config_module.html\">https:\/\/docs.ansible.com\/ansible\/latest\/modules\/eos_config_module.html<\/a><\/p>\n\n\n\n<p>After some time, I managed to fix the playbook and my environment and I could run the playbook using the ssh connector (but I was ignoring a warning about &#8220;provider&#8221; not needed&#8230;)<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">(testdir2)\/ansible master$ cat ansible-hosts\n[all:vars]\nansible_python_interpreter=\/home\/tomas\/storage\/technology\/arista\/testdir2\/bin\/python\nansible_user='tomas'\nansible_password='tomas123'\n[ceoslab]\nr1 ansible_host=127.0.0.1 ansible_port=2000\nr2 ansible_host=127.0.0.1 ansible_port=2001\nr3 ansible_host=127.0.0.1 ansible_port=2002\n\n(testdir2)\/ansible master$ cat group_vars\/ceoslab.yaml\nansible_network_os: eos\n<\/pre>\n\n\n\n<p>The output:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/ansible master$ ansible-playbook playbooks\/collect-facts-cli.yaml\nPLAY [Run commands on ceos lab]\nTASK [Collect all facts from device] ***\n[WARNING]: provider is unnecessary when using network_cli and will be ignored\n[WARNING]: default value for <code>gather_subset<\/code> will be changed to <code>min<\/code> from <code>!config<\/code> v2.11 onwards\nok: [r1]\nok: [r3]\nok: [r2]\nTASK [Display result] ****\nok: [r2] => {\n\"msg\": \"Model is cEOSLab and it is running 4.23.3M\"\n}\nok: [r1] => {\n\"msg\": \"Model is cEOSLab and it is running 4.23.3M\"\n}\nok: [r3] => {\n\"msg\": \"Model is cEOSLab and it is running 4.23.3M\"\n}\nPLAY RECAP *\nr1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0\nr2 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0\nr3 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0<\/pre>\n\n\n\n<p>Ok, so getting the playbook using the API shouldnt be that difficult? It was.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">The full traceback is:\nFile \"\/tmp\/ansible_eos_facts_payload_vz7c7ipu\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/network\/common\/network.py\", line 229, in get_capabilities\ncapabilities = Connection(module._socket_path).get_capabilities()\nFile \"\/tmp\/ansible_eos_facts_payload_vz7c7ipu\/ansible_eos_facts_payload.zip\/ansible\/module_utils\/connection.py\", line 185, in <strong>rpc<\/strong>\nraise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)\nfatal: [r1]: FAILED! => {\n\"changed\": false,\n\"invocation\": {\n\"module_args\": {\n\"auth_pass\": null,\n\"authorize\": null,\n\"gather_network_resources\": null,\n\"gather_subset\": [\n\"all\"\n],\n\"host\": null,\n\"password\": null,\n\"port\": null,\n\"provider\": null,\n\"ssh_keyfile\": null,\n\"timeout\": null,\n\"transport\": null,\n\"use_ssl\": null,\n\"username\": null,\n\"validate_certs\": null\n}\n},\n\"msg\": \"Could not connect to http:\/\/127.0.0.1:80\/command-api: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1056)\"\n}<\/pre>\n\n\n\n<p>I was surprised that it was using port 80. I was pretty sure I was providing the correct port (900x) so somehow my data wasnt being processed.<\/p>\n\n\n\n<p>I wasn&#8217;t clearly paying attention to the documentation:<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/modules\/eos_facts_module.html#eos-facts-module\">https:\/\/docs.ansible.com\/ansible\/latest\/modules\/eos_facts_module.html#eos-facts-module<\/a><\/p>\n\n\n\n<p>It says clearly that &#8220;provider&#8221; is deprecated since 2.5! I am using 2.9<\/p>\n\n\n\n<p>As well, I have a very poor knowledge of ansible and I didnt understand the concept of &#8220;connection&#8221;. The SSH was using &#8220;network_cli&#8221; and API was using &#8220;httpapi&#8221;.<\/p>\n\n\n\n<p>I was very close to give up the API connection when somehow I searched for &#8220;ansible network_cli&#8221; and I found documentation for that plugging. Then I searched for &#8220;httpapi&#8221; and it was gold!<\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/plugins\/connection\/network_cli.html\">https:\/\/docs.ansible.com\/ansible\/latest\/plugins\/connection\/network_cli.html<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/docs.ansible.com\/ansible\/latest\/plugins\/connection\/httpapi.html\">https:\/\/docs.ansible.com\/ansible\/latest\/plugins\/connection\/httpapi.html<\/a><\/p>\n\n\n\n<p>I realised that I need to pass specific vars for getting the HTTPS connection. So at the end, managed to get the config right for both SSH and HTTPS:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/ansible master$ cat ansible-hosts\n[all:vars]\nansible_python_interpreter=\/home\/tomas\/storage\/technology\/arista\/testdir2\/bin\/python\nansible_user='tomas'\nansible_password='tomas123'\n[ceoslab]\nr1 ansible_host=127.0.0.1 ansible_port=2000 ansible_httpapi_port=9000\nr2 ansible_host=127.0.0.1 ansible_port=2001 ansible_httpapi_port=9001\nr3 ansible_host=127.0.0.1 ansible_port=2002 ansible_httpapi_port=9002\n\n\/ansible master$ cat group_vars\/ceoslab.yaml\nansible_network_os: eos\n#start - eapi config\nansible_httpapi_use_ssl: 'yes'\nansible_httpapi_validate_certs: 'no'\nansible_httpapi_password: \"{{ ansible_password }}\"\n#end - eapi config<\/pre>\n\n\n\n<p>The output:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">ansible master$ ansible-playbook playbooks\/collect-facts-eapi.yaml\nPLAY [Run commands on remote ceos lab] *\nTASK [Collect all facts from device] ***\n[WARNING]: default value for <code>gather_subset<\/code> will be changed to <code>min<\/code> from <code>!config<\/code> v2.11 onwards\nok: [r3]\nok: [r1]\nok: [r2]\nTASK [Display result] ****\nok: [r2] => {\n\"msg\": \"Model is cEOSLab and it is running 4.23.3M\"\n}\nok: [r1] => {\n\"msg\": \"Model is cEOSLab and it is running 4.23.3M\"\n}\nok: [r3] => {\n\"msg\": \"Model is cEOSLab and it is running 4.23.3M\"\n}\nPLAY RECAP *\nr1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0\nr2 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0\nr3 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0<\/pre>\n\n\n\n<p>At the end of the day, the scripts are identical apart from the &#8220;connection&#8221; var:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">\/ansible\/playbooks master$ diff collect-facts-cli.yaml collect-facts-eapi.yaml\n4c4\n&lt; connection: network_cli\n---\n> connection: httpapi<\/pre>\n\n\n\n<p>I think you can pass a var to the playbook via the CLI so I will try later.<\/p>\n\n\n\n<p>My recommendation is always to use &#8220;-vvv&#8221;.<\/p>\n\n\n\n<p>BTW, A good ansible summary I found:<\/p>\n\n\n\n<p><a href=\"https:\/\/gist.github.com\/andreicristianpetcu\/b892338de279af9dac067891579cad7d\">https:\/\/gist.github.com\/andreicristianpetcu\/b892338de279af9dac067891579cad7d<\/a><\/p>\n\n\n\n<p>In summary, I found ansible more difficult to troubleshoot that nornir. In nornir, is pure python, I can run ipdb wherever a I want.<\/p>\n\n\n\n<p>But anyway, I learned things. I will add try to write a bit more complex playbooks.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A couple of years a go I wrote a playbook with ansible to use napalm for configuring some switches. I wanted to test again ansible as I am quite rusty and there is always demand for that. I started with just something basic with my ceos lab. All my code is here: https:\/\/github.com\/thomarite\/ceos-testing\/tree\/master\/ansible Initially I &hellip; <a href=\"https:\/\/blog.thomarite.uk\/index.php\/2020\/07\/09\/ansible-troubleshooting\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Ansible &#8211; Troubleshooting&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[],"class_list":["post-270","post","type-post","status-publish","format-standard","hentry","category-automation"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/270","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=270"}],"version-history":[{"count":1,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/270\/revisions"}],"predecessor-version":[{"id":271,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/270\/revisions\/271"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=270"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=270"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=270"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}