Kata环境搭建1 —— kata+containerd

Kata + containerd

environment

ubuntu 22.04 (刚开始用20.04，编译的时候musl出现一些问题)

Build and install Kata Containers

install neccesary tools

需要注意，先git clone kata-conatneir的代码之后再安装以下软件，并且一定要按照katacontainer/version.yaml中的对应版本进行下载，不然后期编译可能会出现很多难解决的问题。

golang：安装之后添加系统路径
rust：可以使用kata-containers/ci/install_rust.sh下载
make
gcc

Build and install the Kata Containers runtime

$ git clone https://github.com/kata-containers/kata-containers.git
$ git checkout 3.1.0-alpha1
$ pushd kata-containers/src/runtime
$ make && sudo -E "PATH=$PATH" make install
$ sudo mkdir -p /etc/kata-containers/
$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
$ popd

Configure to use rootfs image

$ sudo sed -i 's/^\(initrd =.*\)/# \1/g' /etc/kata-containers/configuration.toml # uncommented congiguration file

Enable seccomp

$ sudo sed -i '/^disable_guest_seccomp/ s/true/false/' /etc/kata-containers/configuration.toml

Enable full debug

$ sudo sed -i -e 's/^# *\(enable_debug\).*=.*$/\1 = true/g' /etc/kata-containers/configuration.toml
$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.log=debug initcall_debug"/g' /etc/kata-containers/configuration.toml

Create and install rootfs and initrd image

Build a custom Kata agent

You should only do this step if you are testing with the latest version of the agent

configure default libc(musl)

$ export ARCH="$(uname -m)"
$ if [ "$ARCH" = "ppc64le" -o "$ARCH" = "s390x" ]; then export LIBC=gnu; else export LIBC=musl; fi
$ [ "${ARCH}" == "ppc64le" ] && export ARCH=powerpc64le
$ rustup target add "${ARCH}-unknown-linux-${LIBC}"

build the agent

$ make -C kata-containers/src/agent

在这步可能会出错：
/usr/bin/ld: /home/niteesh/github/kata-containers/src/agent/target/x86_64-unknown-linux-musl/release/deps/kata_agent-46b61c92992321dd.kata_agent.c2ef787b-cgu.3.rcgu.o: undefined reference to symbol 'pthread_mutex_lock@@GLIBC_2.2.5'
/usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
此处错误需要安装libseccomp并将其静态链接到musl上。解决方法：
# Install libseccomp for static linking
sudo -E PATH=$PATH GOPATH=$GOPATH kata-containers/ci/install_libseccomp.sh /tmp/kata-libseccomp /tmp/kata-gperf
export LIBSECCOMP_LINK_TYPE=static
export LIBSECCOMP_LIB_PATH=/tmp/kata-libseccomp/lib
需要注意rust和版本一定要和version.yaml一致，否则很大可能报错

来源参考：

https://github.com/kata-containers/kata-containers/issues/5044

Create a rootfs image

Create a local rootfs

install docker

run rootfs.sh

$ export distro="ubuntu" # example
$ export ROOTFS_DIR="$(realpath kata-containers/tools/osbuilder/rootfs-builder/rootfs)"
$ sudo rm -rf "${ROOTFS_DIR}"
$ pushd kata-containers/tools/osbuilder/rootfs-builder
$ script -fec 'sudo -E USE_DOCKER=true ./rootfs.sh "${distro}"'
$ popd

Add a custom agent to the image

You should only do this step if you are testing with the latest version of the agent

$ sudo install -o root -g root -m 0550 -t "${ROOTFS_DIR}/usr/bin" "${ROOTFS_DIR}/../../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent"
$ sudo install -o root -g root -m 0440 "${ROOTFS_DIR}/../../../../src/agent/kata-agent.service" "${ROOTFS_DIR}/usr/lib/systemd/system/"
$ sudo install -o root -g root -m 0440 "${ROOTFS_DIR}/../../../../src/agent/kata-containers.target" "${ROOTFS_DIR}/usr/lib/systemd/system/"

Build a rootfs image

$ pushd  kata-containers/tools/osbuilder/image-builder
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh "${ROOTFS_DIR}"'
$ popd

Install the rootfs image

$ pushd kata-containers/tools/osbuilder/image-builder
$ commit="$(git log --format=%h -1 HEAD)"
$ date="$(date +%Y-%m-%d-%T.%N%z)"
$ image="kata-containers-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)
$ popd

Install guest kernel images

Setup kernel source code

$ git clone github.com/kata-containers/kata-containers
$ cd kata-containers/tools/packaging/kernel
$ ./build-kernel.sh setup

Build the kernel

$ ./build-kernel.sh build

在build的过程中可能会缺少一些头文件，如xxx.h，解决方法是安装相应的库，通常库名为libxxx-dev，即：
$ sudo apt-get install libxxx-dev

Install the Kernel in the default path for Kata

$ sudo ./build-kernel.sh install

Install a hypervisor

Build a custom QEMU

get qemu version suitable to kata

$ source kata-containers/tools/packaging/scripts/lib.sh
$ qemu_version="$(get_from_kata_deps "assets.hypervisor.qemu.version")"
$ echo "${qemu_version}"

Get source from the matching branch of QEMU

$ git clone -b "${qemu_version}" https://github.com/qemu/qemu.git
$ your_qemu_directory="$(realpath qemu)"

manage the build and packaging of QEMU

$ packaging_dir="$(realpath kata-containers/tools/packaging)"
$ "$packaging_dir/scripts/apply_patches.sh" "$packaging_dir/qemu/patches/5.2.x/" #apply patch

build utilizing the same options as Kata

$ pushd "$your_qemu_directory"
$ "$packaging_dir/scripts/configure-hypervisor.sh" kata-qemu > kata.cfg
$ eval ./configure "$(cat kata.cfg)"
$ make -j $(nproc --ignore=1)
# Optional
$ sudo -E make install
$ popd

Build virtiofsd

When using the file system type virtio-fs (default), virtiofsd is required

$ pushd kata-containers/tools/packaging/static-build/virtiofsd
$ ./build.sh
$ popd

Modify /etc/kata-containers/configuration.toml and update value virtio_fs_daemon = "/path/to/kata-containers/tools/packaging/static-build/virtiofsd/virtiofsd/virtiofsd" to point to the binary.

Check hardware requirements

If your system is not able to run Kata Containers, the command will error out and explain why:

$ sudo kata-runtime check

error ：kernel property kvm_amd not found

solution：此处需要注意，如果是vmware下运行Ubuntu系统需要在vmware设置里将虚拟化引擎打开：

同时主机的hyper-V要关闭，因为Vmware与Hyper-v不兼容，否则会导致报错(如"此平台不支持虚拟化的AMD-V/RVI")，解决方案参考：

[关于“ VMware Workstation 16 此平台不支持虚拟化的Intel VT-x/EPT. 不使用虚拟化的Intel VT-x/EPT,是否继续？”的有关问题的总结解答]

Run Kata Containers with Containerd

Install

install containerd

可以下载二进制文件解压，也可以用apt直接安装。我使用的是apt安装的方式，和下载二进制文件不同的是，还需要自己安装CNI

install CNI plugins

$ git clone https://github.com/containernetworking/plugins.git
$ pushd plugins
$ ./build_linux.sh
$ sudo mkdir /opt/cni
$ sudo cp -r bin /opt/cni/
$ popd

install cri-tools

$ git clone https://github.com/kubernetes-sigs/cri-tools.git
$ pushd cri-tools
$ make
$ sudo -E make install
$ popd

Configuration

Configure containerd to use Kata Containers

# /etc/containerd/config.toml
[plugins]
  [plugins.cri]
    [plugins.cri.cni]
      # conf_dir is the directory in which the admin places a CNI conf.
      conf_dir = "/etc/cni/net.d"
    [plugins.cri.containerd]
      no_pivot = false
    [plugins.cri.containerd.runtimes]
      [plugins.cri.containerd.default_runtime]
         runtime_type = "io.containerd.runtime.v1.linux"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
         privileged_without_host_devices = false
         runtime_type = "io.containerd.runc.v2"
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
      [plugins.cri.containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
         privileged_without_host_devices = true
         pod_annotations = ["io.katacontainers.*"]
         container_annotations = ["io.katacontainers.*"]
         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
            ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml"

add the CNI configuration in the containerd configuration

$ mkdir -p /etc/cni/net.d
$ cat >/etc/cni/net.d/10-mynet.conf <<EOF
{
    "cniVersion": "0.2.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.1.0.0/16",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}
EOF

reference the configuration directory through containerd /etc/containerd/onfig.toml

[plugins.cri.cni]
    # conf_dir is the directory in which the admin places a CNI conf.
    conf_dir = "/etc/cni/net.d"

The configuration file of crictl command line tool in cri-tools locates at /etc/crictl.yaml

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true

Run

$ sudo ctr image pull docker.io/library/busybox:latest
$ sudo ctr run --cni --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

posted @ 2023-02-10 10:34 ppddcsz 阅读(721) 评论(0) 编辑收藏举报

刷新页面返回顶部

ppddcsz