[Docker] Storing Container Data In Docker Volumes
Storing data within a container image is one option for automating a container with data, but it requires a copy of the data to be in each container you run. For static files, this can be a waste of resources. Each file might not amount to more than a few megabytes, but once the containers are scaled up to handle a production load, that few megabytes can turn into gigabytes of waste. Instead, you can store one copy of the static files in a Docker volume for easy sharing between containers. In this lab, you will learn how Docker volumes interact with containers. You will do this by creating new volumes and attaching them to containers. You'll then clean up space left by anonymous volumes created automatically by the containers. Finally, you'll learn about backup strategies for your volumes.
### Check the setup
To start with, we have already fews images downloaded from DockerHub, and we currently don't have any volume
[cloud_user@ip-10-0-1-129 ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
httpd 2.4 6e794a483258 7 days ago 145MB
bash latest 2552d69c6348 5 weeks ago 12.9MB
postgres 12.1 cf879a45faaa 2 years ago 394MB
[cloud_user@ip-10-0-1-129 ~]$ docker volume ls
DRIVER VOLUME NAME
Anonymous volume
Run the postgres image:
docker run --name db1 -d postgres:12.1
docker run --name db1 -d postgres:12.1
Then let's check volume:
[cloud_user@ip-10-0-1-129 ~]$ docker volume ls
DRIVER VOLUME NAME
local 45481ab743b77a1ce8bd3f480b50172e5f4d1c31ae51b565a0f371b11d14100d
local affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199
As we can see that, when postgres start, it needs some space to store the data, so it creates anymonus volume to do that.
Now, let's take a close look to the volume:
- we want to
inspectdb1's volume - since the inspect response is large, we only interest in
Mountsinformation:-f '{{ json .Mounts }}' - then we use
python -m json.toolto pretty the json
[cloud_user@ip-10-0-1-129 ~]$ docker inspect db1 -f '{{ json .Mounts }}' | python -m json.tool
[
{
"Destination": "/var/lib/postgresql/data",
"Driver": "local",
"Mode": "",
"Name": "affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199",
"Propagation": "",
"RW": true,
"Source": "/var/lib/docker/volumes/affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199/_data",
"Type": "volume"
}
]
As you can see, the Name: affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199 is the same as our volumes ouput.
--rm flag, clear the container after stop the container
Now let's run a third container with --rm flag
[cloud_user@ip-10-0-1-129 ~]$ docker run --name dbTmp --rm -d postgres:12.1
1956babce36e413f1402400cbe3201f3a88c5748c82cfa18654dccc1efffe6c9
[cloud_user@ip-10-0-1-129 ~]$ docker volume ls
DRIVER VOLUME NAME
local 6f8c3d724bcd6c81738d6e2f64bfa65edcfc84170a10ae76ddd9c9ba7015b905
local 45481ab743b77a1ce8bd3f480b50172e5f4d1c31ae51b565a0f371b11d14100d
local affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199
Let's stop the containers
[cloud_user@ip-10-0-1-129 ~]$ docker stop db2 dbTmp
db2
dbTmp
[cloud_user@ip-10-0-1-129 ~]$ docker volume ls
DRIVER VOLUME NAME
local 45481ab743b77a1ce8bd3f480b50172e5f4d1c31ae51b565a0f371b11d14100d
local affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199
As you can see, after stop db2, dbTmp, we still have two volume, one for db1, another is for db2.
The volume for dbTmp has been removed with container by --rm flag when container stopped.
Create Docker Volume
Let's create a docker volume to put our website code:
docker volume create webstie
[cloud_user@ip-10-0-1-129 ~]$ docker volume ls
DRIVER VOLUME NAME
local 45481ab743b77a1ce8bd3f480b50172e5f4d1c31ae51b565a0f371b11d14100d
local affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199
local website
Let's copy our website code from /home/cloud_user/widget-factory-inc/web/* to volume /var/lib/docker/volumes/website/_data/
sudo cp -r /home/cloud_user/widget-factory-inc/web/* /var/lib/docker/volumes/website/_data/
verify the files is there:
[cloud_user@ip-10-0-1-129 ~]$ sudo ls -l /var/lib/docker/volumes/website/_data/
total 16
drwxr-xr-x. 2 root root 76 Jan 25 14:23 img
-rw-r--r--. 1 root root 3059 Jan 25 14:23 index.html
-rw-r--r--. 1 root root 2910 Jan 25 14:23 quote.html
-rw-r--r--. 1 root root 2611 Jan 25 14:23 support.html
-rw-r--r--. 1 root root 2645 Jan 25 14:23 widgets.html
Serve the website using httpd
To server the website using httpd and mount website volume to it
docker run --name web1 -d -p 80:80 -v website:/usr/local/apache2/htdocs:ro httpd:2.4
We should be able to see the site.
Now, run with the --rm flag, see what's happen:
docker run --name webTmp -d --rm -v website:/usr/local/apache2/htdocs:ro httpd:2.4
[cloud_user@ip-10-0-1-129 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
34ac79c977f2 httpd:2.4 "httpd-foreground" 2 seconds ago Up 1 second 80/tcp webTmp
f2e2aa007625 httpd:2.4 "httpd-foreground" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp web1
f08726cd61cc postgres:12.1 "docker-entrypoint.s…" 23 minutes ago Up 23 minutes 5432/tcp db1
Let's stop the webTmp
[cloud_user@ip-10-0-1-129 ~]$ docker stop webTmp
webTmp
[cloud_user@ip-10-0-1-129 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2e2aa007625 httpd:2.4 "httpd-foreground" 5 minutes ago Up 5 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp web1
f08726cd61cc postgres:12.1 "docker-entrypoint.s…" 26 minutes ago Up 26 minutes 5432/tcp db1
[cloud_user@ip-10-0-1-129 ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2e2aa007625 httpd:2.4 "httpd-foreground" 5 minutes ago Up 5 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp web1
c00a4b92b0c8 postgres:12.1 "docker-entrypoint.s…" 25 minutes ago Exited (0) 13 minutes ago db2
f08726cd61cc postgres:12.1 "docker-entrypoint.s…" 26 minutes ago Up 26 minutes 5432/tcp db1
As we can see, webTmp has gone.
Check the volumes:
[cloud_user@ip-10-0-1-129 ~]$ docker volume ls
DRIVER VOLUME NAME
local 45481ab743b77a1ce8bd3f480b50172e5f4d1c31ae51b565a0f371b11d14100d
local affa07409189939e60eb3d520092fac428ac9cde54a63d5b4311b4f3694d9199
local website
Then we want to prune the volumes:
[cloud_user@ip-10-0-1-129 ~]$ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
Actually no volume has been removed! It is because even we stop db2, but the container is still there, so docker won't delete the volume which has link to a container
[cloud_user@ip-10-0-1-129 ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2e2aa007625 httpd:2.4 "httpd-foreground" 8 minutes ago Up 8 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp web1
c00a4b92b0c8 postgres:12.1 "docker-entrypoint.s…" 29 minutes ago Exited (0) 17 minutes ago db2
f08726cd61cc postgres:12.1 "docker-entrypoint.s…" 30 minutes ago Up 30 minutes 5432/tcp db1
Let's remove db2 container
[cloud_user@ip-10-0-1-129 ~]$ docker rm db2
db2
[cloud_user@ip-10-0-1-129 ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2e2aa007625 httpd:2.4 "httpd-foreground" 10 minutes ago Up 10 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp web1
f08726cd61cc postgres:12.1 "docker-entrypoint.s…" 31 minutes ago Up 31 minutes 5432/tcp db1
Now db2 container has been deleted.
Let's remove the volme again:
[cloud_user@ip-10-0-1-129 ~]$ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
45481ab743b77a1ce8bd3f480b50172e5f4d1c31ae51b565a0f371b11d14100d
Total reclaimed space: 41.43MB
Deletion success!
Backup and restore data to a docker volume
Modify volume need sudo
[cloud_user@ip-10-0-1-129 ~]$ sudo su -
[sudo] password for cloud_user:
[root@ip-10-0-1-129 ~]#
First, let's where docker volume is:
[root@ip-10-0-1-129 ~]# docker volume inspect website
[
{
"CreatedAt": "2023-01-25T14:23:53-05:00",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/website/_data",
"Name": "website",
"Options": {},
"Scope": "local"
}
]
Create a zip file of the voluem data
tar czf /tmp/website_$(date +%Y-%m-%d-%H%M).tgzcreate a zip under/tmpfolder with date in name-C /var/lib/docker/volumes/website/_data-C /var/lib/docker/volumes/website/_data .: want to traget directory/var/lib/docker/volumes/website/_dataand.we want to zip everythings in this folder
tar czf /tmp/website_$(date +%Y-%m-%d-%H%M).tgz -C /var/lib/docker/volumes/website/_data .
Let's see what have been generated:
[root@ip-10-0-1-129 ~]# ls -l /tmp/website_*.tgz
-rw-r--r--. 1 root root 11363 Jan 25 14:47 /tmp/website_2023-01-25-1447.tgz
Let's also see the files:
[root@ip-10-0-1-129 ~]# ls -l /tmp/website_*.tgz
-rw-r--r--. 1 root root 11363 Jan 25 14:47 /tmp/website_2023-01-25-1447.tgz
[root@ip-10-0-1-129 ~]# tar tf /tmp/website_2023-01-25-1447.tgz
./
./img/
./img/LargeWidget.png
./img/MediumWidget.png
./img/SmallWidget.png
./index.html
./quote.html
./support.html
./widgets.html
We have everthing backup! But one thing is not ideal is that, we don't want everytime to use root user to backup data.
exit
Create a backup with non root access
- run a container with volumen
websiteto container/websitefolder - create a volume with
/tmp(where zip files live) to container/backup - zip everything under container's
/websitefolder to/backupfolder
docker run -it --rm -v website:/website -v /tmp:/backup bash tar czf /backup/website_$(date +%Y-%m-%d-%H-%M).tgz -C /website .
We should be able to see the zip files:
[cloud_user@ip-10-0-1-86 ~]$ ls -l /tmp
total 1212
-rw-r--r--. 1 root root 1212535 Jan 18 2022 aws-cfn-bootstrap-latest.amzn1.noarch.rpm
drwx------. 3 root root 17 Jan 26 02:30 systemd-private-65fe0c5b569d4f54bfac42ae4f2269c1-chronyd.service-xzrmML
-rw-r--r--. 1 root root 11364 Jan 26 02:52 website_2023-01-26-0252.tgz
-rw-r--r--. 1 root root 11351 Jan 26 02:55 website_2023-01-26-0255.tgz
Restore the files
To test our backup we need to use root access again: sudo su -
Go to the docker volume folder: cd /var/lib/docker/volumes/
Then cd into website volume folder: cd website/_data/
We want to remove everything: rm -rf *
Make sure everything has been deleted:
[root@ip-10-0-1-86 _data]# ls -l
total 0
Now we want to unzip the zip file /tmp/website_2023-01-26-0255.tgz into the current folder to restore the data
[root@ip-10-0-1-86 _data]# tar xf /tmp/website_2023-01-26-0255.tgz .
[root@ip-10-0-1-86 _data]# ls -l
total 16
drwxr-xr-x. 2 root root 76 Jan 26 02:46 img
-rw-r--r--. 1 root root 3059 Jan 26 02:46 index.html
-rw-r--r--. 1 root root 2910 Jan 26 02:46 quote.html
-rw-r--r--. 1 root root 2611 Jan 26 02:46 support.html
-rw-r--r--. 1 root root 2645 Jan 26 02:46 widgets.html
OK! now we got our data back
