Docker Daemon Start Failure

A strange issue happened without a clear cause and by the time I didn’t fully understand why/how the solution worked, documented here for revisit if necessary.

Note: Run docker command with sudo is not encouraged, add user to docker group is preferred.

Issue: Docker daemon startup fails contineously, checking deamon status and journal:

× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-02-28 04:42:32 UTC; 94ms ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 6747 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 6747 (code=exited, status=1/FAILURE)
        CPU: 266ms

Feb 28 04:42:32 chengdolgob.c.googlers.com systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Feb 28 04:42:32 chengdolgob.c.googlers.com systemd[1]: Stopped Docker Application Container Engine.
Feb 28 04:42:32 chengdolgob.c.googlers.com systemd[1]: docker.service: Start request repeated too quickly.
Feb 28 04:42:32 chengdolgob.c.googlers.com systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 28 04:42:32 chengdolgob.c.googlers.com systemd[1]: Failed to start Docker Application Container Engine

First to adjust the docker daemon unit file to disabke restart by commenting out:

#TimeoutSec=0
#RestartSec=10
#Restart=always

Then run systemctl daemon-reload and restart docker daemon to see detailed error message:

...
Feb 28 04:43:46 chengdolgob.c.googlers.com dockerd[7890]: failed to start daemon: Error initializing network controller: list bridge addresses failed: PredefinedLocalScopeDefaultNetworks List: [192.168.11.0/24]: no availabl
...

Check the network interface ip a s, there is no docker0 bridge.

Solution: To fix it, create new docker network bridge, reference ticket:

# delete docker0 bridge
sudo ip link delete docker0

# 192.168.9.1/24 is from docker daemon.json bip field
sudo ip link add name docker0 type bridge
sudo ip addr add dev docker0 192.168.9.1/24
# or default
sudo ip addr add dev docker0 172.17.0.1/16

The inet IP is from /etc/docker/daemon.json bip field, for example:

{
  "data-root": "/usr/local/google/docker",
  "bip": "192.168.9.1/24",
  "default-address-pools": [
    {
      "base": "192.168.11.0/24",
      "size": 24
    }
  ],
  "storage-driver": "overlay2",
  "debug": true,
  "registry-mirrors": ["https://mirror.gcr.io"]
}

See daemon.json attribute description.

Then after bridge is created, restart docker daemon:

sudo systemctl restart docker