{"id":722,"date":"2021-09-11T17:27:28","date_gmt":"2021-09-11T16:27:28","guid":{"rendered":"https:\/\/blog.thomarite.uk\/?p=722"},"modified":"2021-09-11T17:27:28","modified_gmt":"2021-09-11T16:27:28","slug":"containerlab","status":"publish","type":"post","link":"https:\/\/blog.thomarite.uk\/index.php\/2021\/09\/11\/containerlab\/","title":{"rendered":"Containerlab"},"content":{"rendered":"\n

Some months ago I read about containerlab<\/a> (and here<\/a>). It looked like a very simple way to build labs quickly, easily and multi-vendor. I have used in the past gns3 and docker-topo for my labs but somehow I liked the documentation and the idea to try to mix cEOS with FRR images.<\/p>\n\n\n\n

As I have felt more comfortable with the years with Arista and I had some images in my laptop, I installed the software (no mayor issues following the instructions for Debian) and try the example for a cEOS lab<\/a>. <\/p>\n\n\n\n

It didnt work. The containers started but I didnt get to the Arista CLI, just bash CLI and couldnt see anything runing on them… I remembered some Arista specific processes but none was there. In the following weeks, I tried newer cEOS but no luck always stuck in the same point. But at the end, never had enough time (or put the effort and interest) to troubleshoot the problem properly.<\/p>\n\n\n\n

For too many months, I havent had the chance (I can write a post with excuses) to do much tech self-learning (I can write a book of all things I would like to learn), it was easier cooking or reading.<\/p>\n\n\n\n

But finally, this week, talking with a colleague at work, he mentioned containerlab was great and he used it. I commented that I tried and failed. With that, I finally find a bit of interest and time today to give another go.<\/p>\n\n\n\n

Firstly, I made sure I was running the latest containerlab version and my cEOS was recent enough (4.26.0F) and get to basics, check T-H-E logs! <\/p>\n\n\n\n

So one thing I noticed after paying attention to the startup logs, I could see an warning about lack of memory in my laptop. So I closed several applications and tried again. My lab looked stuck in the same point:<\/p>\n\n\n\n

go:1.16.3|py:3.7.3|tomas@athens:~\/storage\/technology\/containerlabs\/ceos$ sudo containerlab deploy --topo ceos-lab1.yaml \nINFO[0000] Parsing & checking topology file: ceos-lab1.yaml \nINFO[0000] Creating lab directory: \/home\/tomas\/storage\/technology\/containerlabs\/ceos\/clab-ceos \nINFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0\/24', IPv6Subnet='2001:172:20:20::\/64', MTU='1500' \nINFO[0000] config file '\/home\/tomas\/storage\/technology\/containerlabs\/ceos\/clab-ceos\/ceos1\/flash\/startup-config' for node 'ceos1' already exists and will not be generated\/reset \nINFO[0000] Creating container: ceos1                    \nINFO[0000] config file '\/home\/tomas\/storage\/technology\/containerlabs\/ceos\/clab-ceos\/ceos2\/flash\/startup-config' for node 'ceos2' already exists and will not be generated\/reset \nINFO[0000] Creating container: ceos2                    \nINFO[0003] Creating virtual wire: ceos1:eth1 <--> ceos2:eth1 \nINFO[0003] Running postdeploy actions for Arista cEOS 'ceos2' node \nINFO[0003] Running postdeploy actions for Arista cEOS 'ceos1' node \n<\/code><\/pre>\n\n\n\n
I did a bit of searching about containerlab and ceos, for example, I could see this blog<\/a> where the author started up successfully a lab with cEOS and I could see his logs!<\/p>\n\n\n\n
So it was clear, my containers were stuck. So I searched for that message “Running postdeploy actions for Arista cEOS”.<\/p>\n\n\n\n
I didnt see anything promising, just links back to the main container lab ceos page<\/a>. I read it again and I noticed something in the bottom of the page regarding a known issue<\/a>….  So I checked if that applied to me (although I doubted as it looked like it was for CentOS…) and indeed it applied to me too!<\/p>\n\n\n\n
$ docker logs clab-ceos-ceos2\nFailed to mount cgroup at \/sys\/fs\/cgroup\/systemd: Operation not permitted<\/code><\/pre>\n\n\n\n
So I started to find info about what is cgroup: link1<\/a>, link2<\/a><\/p>\n\n\n\n
First I wanted to check what cgroup version I was running. With this link<\/a>, I could see that based on my kernel version, I should have cgroup2:<\/p>\n\n\n\n
$ grep cgroup \/proc\/filesystems\nnodev\tcgroup\nnodev\tcgroup2\n\n$ ls \/sys\/fs\/cgroup\/memory\/\ncgroup.clone_children           memory.kmem.tcp.limit_in_bytes      memory.stat\ncgroup.event_control            memory.kmem.tcp.max_usage_in_bytes  memory.swappiness\ncgroup.procs                    memory.kmem.tcp.usage_in_bytes      memory.usage_in_bytes\ncgroup.sane_behavior            memory.kmem.usage_in_bytes          memory.use_hierarchy\ndev-hugepages.mount             memory.limit_in_bytes               notify_on_release\ndev-mqueue.mount                memory.max_usage_in_bytes           proc-fs-nfsd.mount\ndocker                          memory.memsw.failcnt                proc-sys-fs-binfmt_misc.mount\nmachine.slice                   memory.memsw.limit_in_bytes         release_agent\nmemory.failcnt                  memory.memsw.max_usage_in_bytes     sys-fs-fuse-connections.mount\nmemory.force_empty              memory.memsw.usage_in_bytes         sys-kernel-config.mount\nmemory.kmem.failcnt             memory.move_charge_at_immigrate     sys-kernel-debug.mount\nmemory.kmem.limit_in_bytes      memory.numa_stat                    sys-kernel-tracing.mount\nmemory.kmem.max_usage_in_bytes  memory.oom_control                  system.slice\nmemory.kmem.slabinfo            memory.pressure_level               tasks\nmemory.kmem.tcp.failcnt         memory.soft_limit_in_bytes          user.slice\n<\/code><\/pre>\n\n\n\n
As I had “cgroup.*” in my “\/sys\/fs\/cgroup\/memory” it was confirmed I was running cgroup2.<\/p>\n\n\n\n
So how could I change to cgroup1 for docker only?<\/p>\n\n\n\n
It seems that I couldnt change that only for an application because it is parameter that you pass to the kernel in boot time.<\/p>\n\n\n\n
I learned that there is something called podman to replace docker in this blog<\/a>.<\/p>\n\n\n\n
So at the end, searching how to change cgroup in Debian, I used this link<\/a>:<\/p>\n\n\n\n
$ cat \/etc\/default\/grub\n...\n# systemd.unified_cgroup_hierarchy=0 enables cgroupv1 that is needed for containerlabs to run ceos.... \n# https:\/\/github.com\/srl-labs\/containerlab\/issues\/467\n# https:\/\/mbien.dev\/blog\/entry\/java-in-rootless-containers-with\nGRUB_CMDLINE_LINUX_DEFAULT=\"quiet systemd.unified_cgroup_hierarchy=0\"\n....\n$ sudo grub-mkconfig -o \/boot\/grub\/grub.cfg\n....\n$ sudo reboot.<\/code><\/pre>\n\n\n\n
Good thing that the laptop rebooted fine! That was a relief \ud83d\ude42<\/p>\n\n\n\n
Then I checked if the change made any difference. It failed but because it containerlab couldnt connect to docker… somehow docker had died. I restarted again docker and tried container lab…<\/p>\n\n\n\n
$ sudo containerlab deploy --topo ceos-lab1.yaml \nINFO[0000] Parsing & checking topology file: ceos-lab1.yaml \nINFO[0000] Creating lab directory: \/home\/xxx\/storage\/technology\/containerlabs\/ceos\/clab-ceos \nINFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0\/24', IPv6Subnet='2001:172:20:20::\/64', MTU='1500' \nINFO[0000] config file '\/home\/xxx\/storage\/technology\/containerlabs\/ceos\/clab-ceos\/ceos1\/flash\/startup-config' for node 'ceos1' already exists and will not be generated\/reset \nINFO[0000] Creating container: ceos1                    \nINFO[0000] config file '\/home\/xxx\/storage\/technology\/containerlabs\/ceos\/clab-ceos\/ceos2\/flash\/startup-config' for node 'ceos2' already exists and will not be generated\/reset \nINFO[0000] Creating container: ceos2                    \nINFO[0003] Creating virtual wire: ceos1:eth1 <--> ceos2:eth1 \nINFO[0003] Running postdeploy actions for Arista cEOS 'ceos2' node \nINFO[0003] Running postdeploy actions for Arista cEOS 'ceos1' node \nINFO[0145] Adding containerlab host entries to \/etc\/hosts file \n+---+-----------------+--------------+------------------+------+-------+---------+----------------+----------------------+\n| # |      Name       | Container ID |      Image       | Kind | Group |  State  |  IPv4 Address  |     IPv6 Address     |\n+---+-----------------+--------------+------------------+------+-------+---------+----------------+----------------------+\n| 1 | clab-ceos-ceos1 | 2807cd2f689f | ceos-lab:4.26.0F | ceos |       | running | 172.20.20.2\/24 | 2001:172:20:20::2\/64 |\n| 2 | clab-ceos-ceos2 | e5d2aa4578b5 | ceos-lab:4.26.0F | ceos |       | running | 172.20.20.3\/24 | 2001:172:20:20::3\/64 |\n+---+-----------------+--------------+------------------+------+-------+---------+----------------+----------------------+\n$ sudo clab graph -t ceos-lab1.yaml \nINFO[0000] Parsing & checking topology file: ceos-lab1.yaml \nINFO[0000] Listening on :50080...       <\/code><\/pre>\n\n\n\n
After a bit, it seems it worked! And learned about an option to show a graph of your topology with “graph”<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
I checked the ceos container logs<\/p>\n\n\n\n
$ docker logs clab-ceos-ceos1\n....\n[  OK  ] Started SYSV: Eos system init scrip...uns after POST, before ProcMgr).\n         Starting Power-On Self Test...\n         Starting EOS Warmup Service...\n[  OK  ] Started Power-On Self Test.\n[  OK  ] Reached target EOS regular mode.\n[  OK  ] Started EOS Diagnostic Mode.\n[     *] A start job is running for EOS Warmup Service (2min 9s \/ no limit)Reloading.\n$ \n$ docker exec -it clab-ceos-ceos1 Cli\nceos1>\nceos1>enable \nceos1#show version \n cEOSLab\nHardware version: \nSerial number: \nHardware MAC address: 001c.7389.2099\nSystem MAC address: 001c.7389.2099\n\nSoftware image version: 4.26.0F-21792469.4260F (engineering build)\nArchitecture: i686\nInternal build version: 4.26.0F-21792469.4260F\nInternal build ID: c5b41f65-54cd-44b1-b576-b5c48700ee19\n\ncEOS tools version: 1.1\nKernel version: 5.10.0-8-amd64\n\nUptime: 0 minutes\nTotal memory: 8049260 kB\nFree memory: 2469328 kB\n\nceos1#\nceos1#show interfaces description \nInterface                      Status         Protocol           Description\nEt1                            up             up                 \nMa0                            up             up                 \nceos1#show running-config interfaces ethernet 1\ninterface Ethernet1\nceos1#\n<\/code><\/pre>\n\n\n\n
Yes! Finally working!<\/p>\n\n\n\n
So now, I dont have excuses to keep learning new things!<\/p>\n\n\n\n
BTW, these are the different versions I am using at the moment:<\/p>\n\n\n\n
$ uname -a\nLinux athens 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU\/Linux \n\n$ docker -v\nDocker version 20.10.5+dfsg1, build 55c4c88\n\n$ containerlab version\n\n                           _                   _       _     \n                 _        (_)                 | |     | |    \n ____ ___  ____ | |_  ____ _ ____   ____  ____| | ____| | _  \n\/ ___) _ \\|  _ \\|  _)\/ _  | |  _ \\ \/ _  )\/ ___) |\/ _  | || \\ \n( (__| |_|| | | | |_( ( | | | | | ( (\/ \/| |   | ( ( | | |_) )\n\\____)___\/|_| |_|\\___)_||_|_|_| |_|\\____)_|   |_|\\_||_|____\/ \n\n    version: 0.17.0\n     commit: eba1b82\n       date: 2021-08-25T09:31:53Z\n     source: https:\/\/github.com\/srl-labs\/containerlab\n rel. notes: https:\/\/containerlab.srlinux.dev\/rn\/0.17\/\n\n<\/code><\/pre>\n\n\n\n
My concern is, how this cgroup1 will affect other applications like kubernetes<\/a>? <\/p>\n\n\n\n
BTW, I have the same issue with containerlab as with docker-topo, when I use “Alt+Home(left arrow)” my laptop leave X-Windows and gets to the tty!!!<\/p>\n","protected":false},"excerpt":{"rendered":"
Some months ago I read about containerlab (and here). It looked like a very simple way to build labs quickly, easily and multi-vendor. I have used in the past gns3 and docker-topo for my labs but somehow I liked the documentation and the idea to try to mix cEOS with FRR images. As I have … Continue reading “Containerlab”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"class_list":["post-722","post","type-post","status-publish","format-standard","hentry","category-networks"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/722","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=722"}],"version-history":[{"count":1,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/722\/revisions"}],"predecessor-version":[{"id":724,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/722\/revisions\/724"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=722"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=722"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}