How to use Kata Containers with virtio-fs
Update for Kata Containers 1.7 and later
This HowTo is obsolete as of Kata Containers 1.7. virtio-fs has been included in Kata Containers and can be enabled as described in the official Kata Containers documentation. It is no longer necessary to build from virtio-fs repositories since mainline Kata Containers now includes virtio-fs.
Overview
This document describes how to set up Kata Containers with virtio-fs. Container images will be exposed to the sandbox VM using virtio-fs.
Use this guide if you wish to test or benchmark Kata Container workloads. It is easier to develop and debug virtio-fs standalone without Kata due to the smaller number of components involved. A guide for manually running QEMU with virtio-fs is available here.
Kata Containers is an OCI runtime that runs containers inside a virtual machine for better isolation. Docker and Kubernetes (CRI-O) can be configured to launch Kata Containers instead of their default OCI runtimes.
Each VM is called a sandbox. In Kubernetes a sandbox can be thought of the same thing as a pod.
Containers run inside sandboxes. Each sandbox contains an agent process that communicates with the Kata runtime on the host. Commands include creating new containers, executing processes inside containers, querying sandbox information, etc.
Prerequisites
- Fedora 29 host
- Docker Engine installed (dnf install docker)
Components
The following components need to be built:
- A guest kernel with virtio-fs support
- A QEMU with virtio-fs support
- The example virtio-fs daemon (virtiofsd)
- Kata Containers with virtio-fs support
- A Kata Containers initramfs with virtio-fs support
The instructions assume that you already have available a Linux host on which you can build and run the components.
The guest kernel
Note that an upstream Linux 5.4 kernel or later can be used as long as the DAX feature is not used.
On the host, download the virtio-fs kernel tree by:
git clone https://gitlab.com/virtio-fs/linux.git
git checkout virtio-fs-dev
Configure and build this kernel with the following .config file:
wget -O .config https://gitlab.com/virtio-fs/linux/snippets/1846957/raw make -j$(nproc)
Building QEMU
On the host, download and install the virtio-fs QEMU tree by:
git clone https://gitlab.com/virtio-fs/qemu.git cd qemu ./configure --target-list=x86_64-softmmu make -j$(nproc)
and in the same tree, the virtio-fs daemon needs to be built:
make -j$(nproc) virtiofsd
Building Kata Containers
Setting up environment variables
Decide where you want Go to put source code and packages:
export GOPATH=$HOME/go
Decide where you want to build the sandbox root filesystem:
export ROOTFS_DIR=$GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs-ClearLinux
Cloning Kata Containers repositories
Kata consists of several components, each with its own git repository. You need them all.
go get -d -u github.com/kata-containers/runtime go get -d -u github.com/kata-containers/agent go get -d -u github.com/kata-containers/osbuilder go get -d -u github.com/kata-containers/proxy go get -d -u github.com/kata-containers/shim
Ensure you are on Kata 1.6.1:
(cd $GOPATH/src/github.com/kata-containers/agent && git checkout 1.6.1) (cd $GOPATH/src/github.com/kata-containers/osbuilder && git checkout 1.6.1) (cd $GOPATH/src/github.com/kata-containers/proxy && git checkout 1.6.1) (cd $GOPATH/src/github.com/kata-containers/shim && git checkout 1.6.1)
Use the virtio-fs repositories for Kata components that require virtio-fs integration:
cd $GOPATH/src/github.com/kata-containers/runtime
git remote add virtio-fs https://gitlab.com/virtio-fs/runtime.git
git fetch virtio-fs
git checkout virtio-fs/virtio-fs
cd -
Building the runtime
The runtime presents an OCI-compliant runtime interface to Docker and CRI-O. This is where sandbox setup happens and containers are orchestrated.
cd $GOPATH/src/github.com/kata-containers/runtime/ make
Building the proxy
The proxy forwards communications between the host and the sandbox.
cd $GOPATH/src/github.com/kata-containers/proxy make
Building the shim
The shim is a placeholder process on the host that forwards terminal I/O and signals to the actual container process running inside the sandbox.
cd $GOPATH/src/github.com/kata-containers/shim make
Building the Kata Containers initramfs
Build the Fedora root file system:
cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder ./rootfs.sh clearlinux
Now build the sandbox initramfs:
cd $GOPATH/src/github.com/kata-containers/osbuilder/initrd-builder ./initrd_builder.sh $ROOTFS_DIR
Configuring Kata
The Kata configuration file controls the behavior of all Kata components.
wget -O /etc/kata-containers/configuration.toml https://gitlab.com/virtio-fs/runtime/snippets/1846963/raw
Set the following [hypervisor.qemu] variables:
- path - Path to your QEMU binary (e.g. .../qemu/x86_64-softmmu/qemu-system-x86_64)
- kernel - Path to your sandbox kernel (e.g. .../kernel/arch/x86/boot/bzImage)
- initrd - Path to your sandbox initramfs (e.g. .../go/src/github.com/kata-containers/osbuilder/initrd-builder/kata-containers-initrd.img)
- virtio_fs_daemon - Path to your qemu build virtiofsd binary
Set the following [proxy.kata] variables:
- path - Path to your kata-proxy binary (e.g. .../go/src/github.com/kata-containers/proxy/kata-proxy)
Set the following [shim.kata] variables:
- path - Path to your kata-shim binary (e.g. .../go/src/github.com/kata-containers/shim/kata-shim)
Also set all enable_debug variables to true for verbose output.
Add the Kata OCI runtime to Docker:
# mkdir -p /etc/systemd/system/docker.service.d
# cat >/etc/systemd/system/docker.service.d/kata-containers.conf
[Service]
Type=simple
ExecStart=
ExecStart=/usr/bin/dockerd-current -D \
--add-runtime oci=/usr/libexec/docker/docker-runc-current \
--add-runtime kata-runtime=YOUR_GOPATH/src/github.com/kata-containers/runtime/kata-runtime \
--default-runtime=oci \
--containerd /run/containerd.sock \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
--init-path=/usr/libexec/docker/docker-init-current \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY \
$REGISTRIES
^D
# sed -i "s%YOUR_GOPATH%$GOPATH%g" /etc/systemd/system/docker.service.d/kata-containers.conf
# systemctl daemon-reload
# systemctl restart docker
Ensure there are enough hugepages available on the host for the Kata sandbox VM. The default sandbox RAM size is 2G, so reserve 1024 * 2MB hugepages. This is necessary since QEMU is launched with the memory options (share=on) required by vhost-user.
sysctl vm.nr_hugepages=1024
Running containers
Launch a container:
docker run --runtime kata-runtime -it busybox sh
