I rebooted my laptop today and realised that docker wasnt running… It was running before the reboot and I didn’t upgrade anything related to docker (or I thought)
$ docker ps -a Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? $
Let’s check status and start if needed:
root@athens:/var/log# service docker status ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-08-21 08:34:03 BST; 7min ago TriggeredBy: ● docker.socket Docs: https://docs.docker.com Main PID: 12015 (code=exited, status=1/FAILURE) Aug 21 08:34:03 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3. Aug 21 08:34:03 athens systemd[1]: Stopped Docker Application Container Engine. Aug 21 08:34:03 athens systemd[1]: docker.service: Start request repeated too quickly. Aug 21 08:34:03 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:34:03 athens systemd[1]: Failed to start Docker Application Container Engine. Aug 21 08:34:42 athens systemd[1]: docker.service: Start request repeated too quickly. Aug 21 08:34:42 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:34:42 athens systemd[1]: Failed to start Docker Application Container Engine. root@athens:/var/log# root@athens:/var/log# root@athens:/var/log# service docker start Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details. root@athens:/var/log# systemctl status docker.service ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-08-21 08:41:20 BST; 5s ago TriggeredBy: ● docker.socket Docs: https://docs.docker.com Process: 35305 ExecStart=/usr/sbin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE) Main PID: 35305 (code=exited, status=1/FAILURE) Aug 21 08:41:19 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Aug 21 08:41:19 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:41:19 athens systemd[1]: Failed to start Docker Application Container Engine. Aug 21 08:41:20 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3. Aug 21 08:41:20 athens systemd[1]: Stopped Docker Application Container Engine. Aug 21 08:41:20 athens systemd[1]: docker.service: Start request repeated too quickly. Aug 21 08:41:20 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:41:20 athens systemd[1]: Failed to start Docker Application Container Engine. root@athens:/var/log#
Ok, so not much info… let check the recommend details:
root@athens:/var/log# journalctl -xe ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit docker.socket has begun execution. ░░ ░░ The job identifier is 4236. Aug 21 08:41:20 athens systemd[1]: Listening on Docker Socket for the API. ░░ Subject: A start job for unit docker.socket has finished successfully ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit docker.socket has finished successfully. ░░ ░░ The job identifier is 4236. Aug 21 08:41:20 athens systemd[1]: docker.service: Start request repeated too quickly. Aug 21 08:41:20 athens systemd[1]: docker.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ The unit docker.service has entered the 'failed' state with result 'exit-code'. Aug 21 08:41:20 athens systemd[1]: Failed to start Docker Application Container Engine. ░░ Subject: A start job for unit docker.service has failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ A start job for unit docker.service has finished with a failure. ░░ ░░ The job identifier is 4113 and the job result is failed. Aug 21 08:41:20 athens systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://www.debian.org/support ░░ ░░ The unit docker.socket has entered the 'failed' state with result 'service-start-limit-hit'. root@athens:/var/log# systemctl status docker.service log Unit log.service could not be found. ● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-08-21 08:41:20 BST; 1min 2s ago TriggeredBy: ● docker.socket Docs: https://docs.docker.com Process: 35305 ExecStart=/usr/sbin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE) Main PID: 35305 (code=exited, status=1/FAILURE) Aug 21 08:41:19 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Aug 21 08:41:19 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:41:19 athens systemd[1]: Failed to start Docker Application Container Engine. Aug 21 08:41:20 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3. Aug 21 08:41:20 athens systemd[1]: Stopped Docker Application Container Engine. Aug 21 08:41:20 athens systemd[1]: docker.service: Start request repeated too quickly. Aug 21 08:41:20 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:41:20 athens systemd[1]: Failed to start Docker Application Container Engine. root@athens:/var/log#
So “journalctl -xe” and “systemctl status docker.service log” gave nothing useful….
So I searched for “docker.socket: Failed with result ‘service-start-limit-hit'” as it was the message that looked more suspicious. I landed here and tried one command to get more logs that I didnt know: “journaltctl -fu docker”
root@athens:/var/log# journalctl -fu docker -- Logs begin at Sun 2020-02-02 21:12:23 GMT. -- Aug 21 08:42:41 athens dockerd[35469]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaStat Aug 21 08:42:41 athens dockerd[35469]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaEntry Aug 21 08:42:41 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Aug 21 08:42:41 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:42:41 athens systemd[1]: Failed to start Docker Application Container Engine. Aug 21 08:42:41 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 3. Aug 21 08:42:41 athens systemd[1]: Stopped Docker Application Container Engine. Aug 21 08:42:41 athens systemd[1]: docker.service: Start request repeated too quickly. Aug 21 08:42:41 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:42:41 athens systemd[1]: Failed to start Docker Application Container Engine. Aug 21 08:44:32 athens systemd[1]: Starting Docker Application Container Engine… Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.Metrics Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.HugetlbStat Aug 21 08:44:32 athens dockerd[35538]: unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '"' after object key:value pair Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.PidsStat Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.CPUStat Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.CPUUsage Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.Throttle Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.MemoryStat Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.MemoryEntry Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.BlkIOStat Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.BlkIOEntry Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaStat Aug 21 08:44:32 athens dockerd[35538]: proto: duplicate proto type registered: io.containerd.cgroups.v1.RdmaEntry Aug 21 08:44:32 athens systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Aug 21 08:44:32 athens systemd[1]: docker.service: Failed with result 'exit-code'. Aug 21 08:44:32 athens systemd[1]: Failed to start Docker Application Container Engine. Aug 21 08:44:32 athens systemd[1]: docker.service: Scheduled restart job, restart counter is at 1. Aug 21 08:44:32 athens systemd[1]: Stopped Docker Application Container Engine.
And now, yes, I could see the docker logs properly… and found the culprit and fixed. I am pretty sure the last time I played with “/etc/docker/daemon.json” I restarted docker and it was fine…
Anyway, I learned a new command “journaltctl -fu SERVICE” to troubleshoot services.