Docker in Docker for wandb agent

Intro#

Like portainer, to isolate the wand-agent runtime environment inside container(dockernize) without losing the ability to fully access host resources(commence docker run), we must expose the host docker socket to the wandb-agent container.

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Note: Privillage flag is not necessarily required at this stage.

Self-host#

To slim down the image size, I personally prefer a portable static binary file /docker form portainer-ce:latest image instead of installing by apt install docker.io inside Dockerfile (too much disk space consumption).

FROM python:3.9.19-slim
RUN pip install wandb[launch]
# Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3
COPY docker /bin/docker
# separate two RUN for better layer cache reuse policy
RUN wandb login --relogin --host http://192.168.1.3:8080 local-af185xxxxxxxx
ENTRYPOINT ["wandb", "launch-agent", "-e", "xxxxx", "-q", "test-queue"]

Explore#

In real production, an S6 daemon service may be required to keep agent backend alive?
It seems --restart=always policy is enough, since wandb is already the pid1 process, the entrypoint shares the same lifecycle as container does.

docker run -d --restart always --name wandb-agent --hostname agent-3070 \
           -v /var/run/docker.sock:/var/run/docker.sock wandb-agent:test-queue

The key is to share host docker.sock to the guest.
Note: Lower docker version may not support --gpus params in run, better align with host.