如何在ARM上运行k3s? 窥探k3s启动过程!，内附容器镜像多平台包构建
开始之前
最近在对华为云鲲鹏服务器（一种ARM服务器arm64）运行容器可行性做验证，顺便了解了很多ARM和容器相关的知识。一提到arm运行容器首先想到的是k3s，下面是用k3s快速搭建一个kubernetes系统，运行arm的镜像的案例。
环境准备
arm的环境想到最多的估计就是树莓派，这玩意儿平常用的比较少，成本还是相对比较高，要玩arm的环境推荐华为云的鲲鹏服务器。
环境准备：
- 通过华为云购买一台2u4g的鲲鹏服务器。
- 安装欧拉（一种类似CentOS的系统， openEuler）系统
- 下载k3s-arm64程序（不建议k3s-install.sh安装主要是墙的问题）
节点准备
由于k3s需要使用iptable相关功能，所以需要对系统进行设置，配置方法如下（自行转化）：
---
- name: Set SELinux to disabled state
selinux:
state: disabled
when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']
- name: Enable IPv4 forwarding
sysctl:
name: net.ipv4.ip_forward
value: "1"
state: present
reload: yes
- name: Enable IPv6 forwarding
sysctl:
name: net.ipv6.conf.all.forwarding
value: "1"
state: present
reload: yes
- name: Add br_netfilter to /etc/modules-load.d/
copy:
content: "br_netfilter"
dest: /etc/modules-load.d/br_netfilter.conf
when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']
- name: Load br_netfilter
modprobe:
name: br_netfilter
state: present
when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']
- name: Set bridge-nf-call-iptables (just to be sure)
sysctl:
name: "{{ item }}"
value: "1"
state: present
reload: yes
when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']
loop:
- net.bridge.bridge-nf-call-iptables
- net.bridge.bridge-nf-call-ip6tables
- name: Add /usr/local/bin to sudo secure_path
lineinfile:
line: 'Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin'
regexp: "Defaults(\\s)*secure_path(\\s)*="
state: present
insertafter: EOF
path: /etc/sudoers
validate: 'visudo -cf %s'
when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']
文件来源： k3s-ansible/roles/prereq/tasks/main.yml
启动k3s，观察启动过程
这个步骤很简单，直接上命令
./k3s-arm64 server --disable traefik
来观察启动日志
- 启动api-server
INFO[2020-05-26T19:06:51.465258246+08:00] Starting k3s v1.18.2+k3s1 (698e444a)
INFO[2020-05-26T19:06:51.482543936+08:00] Kine listening on unix://kine.sock
INFO[2020-05-26T19:06:51.686990438+08:00] Active TLS secret (ver=) (count 7): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattl
e.io/cn-172.27.16.180:172.27.16.180 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.
svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:b3257e2631debdf2681f1c0f10956c55d518909c88e3581e6a5d97562ab0
fabf]
INFO[2020-05-26T19:06:51.691943011+08:00] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=No
de,RBAC --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/ranch
er/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/ser
ver/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserv
er.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-a
llowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requesthea
der-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/serve
r/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var
/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
I0526 19:06:51.692400 1698 server.go:682] external host was not specified, using 172.27.16.180
I0526 19:06:51.692649 1698 server.go:166] Version: v1.18.2+k3s1
- 启动
kube-scheduler和kube-controller-manager
INFO[2020-05-26T19:06:57.102712037+08:00] Running kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --p
ort=10251 --secure-port=0
INFO[2020-05-26T19:06:57.104170746+08:00] Running kube-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file
=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kube
config --leader-elect=false --port=10252 --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/
tls/service.key --use-service-account-credentials=true
I0526 19:06:57.107918 1698 controllermanager.go:161] Version: v1.18.2+k3s1
I0526 19:06:57.108605 1698 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
INFO[2020-05-26T19:06:58.129139918+08:00] Running cloud-controller-manager --allocate-node-cidrs=true --allow-untagged-cloud=true --bind-address=127.0.0.1 --cloud-provider=k3s --clu
ster-cidr=10.42.0.0/16 --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m --secure-port=0
Flag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release. A cluster-id will be required on cloud instances.
I0526 19:06:58.134700 1698 controllermanager.go:120] Version: v1.18.2+k3s1
W
- 启动
kube-proxy和kubelet
INFO[2020-05-26T19:06:57.794366512+08:00] Starting k3s.cattle.io/v1, Kind=Addon controller
INFO[2020-05-26T19:06:57.794634665+08:00] Node token is available at /var/lib/rancher/k3s/server/token
INFO[2020-05-26T19:06:57.794673746+08:00] To join node to cluster: k3s agent -s https://172.27.16.180:6443 -t ${NODE_TOKEN}
INFO[2020-05-26T19:06:57.795436726+08:00] Waiting for master node startup: resource name may not be empty
I0526 19:06:57.863811 1698 controller.go:606] quota admission added evaluator for: addons.k3s.cattle.io
INFO[2020-05-26T19:06:57.903238233+08:00] Starting helm.cattle.io/v1, Kind=HelmChart controller
INFO[2020-05-26T19:06:57.903286923+08:00] Starting batch/v1, Kind=Job controller
INFO[2020-05-26T19:06:57.903313164+08:00] Starting /v1, Kind=Node controller
INFO[2020-05-26T19:06:57.903343024+08:00] Starting /v1, Kind=Service controller
INFO[2020-05-26T19:06:57.903385605+08:00] Starting /v1, Kind=Pod controller
INFO[2020-05-26T19:06:57.903411445+08:00] Starting /v1, Kind=Endpoints controller
I0526 19:06:57.921880 1698 controller.go:606] quota admission added evaluator for: deployments.apps
INFO[2020-05-26T19:06:59.097796332+08:00] Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=c
groupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3
s/data/bd19bf78c5988a4ae051b7995298195fce498a99fd094991a90fd12a201fe63d/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=/run/k3s/containerd/
containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=image
fs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ecs-b87a --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig -
-kubelet-cgroups=/systemd/system.slice --node-labels= --read-only-port=0 --resolv-conf=/etc/resolv.conf --runtime-cgroups=/systemd/system.slice --serialize-image-pulls=false --tls-c
ert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation t
imeline before being removed.
INFO[2020-05-26T19:06:59.113169378+08:00] Running kube-proxy --cluster-cidr=10.42.0.0/16 --healthz-bind-address=127.0.0.1 --hostname-override=ecs-b87a --kubeconfig=/var/lib/rancher/
k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables
W0526 19:06:59.113328 1698 server.go:225] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
INFO[2020-05-26T19:06:59.128654056+08:00] waiting for node ecs-b87a: nodes "ecs-b87a" not found
I0526 19:06:59.146328 1698 server.go:413] Version: v1.18.2+k3s1
I
- 启动网络组件flannel
I0526 19:07:12.146803 1698 vxlan.go:121] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0526 19:07:12.193748 1698 flannel.go:78] Wrote subnet file to /run/flannel/subnet.env
I0526 19:07:12.193768 1698 flannel.go:82] Running backend.
I0526 19:07:12.193774 1698 vxlan_network.go:60] watching for new subnet leases
I0526 19:07:12.201828 1698 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0526 19:07:12.201875 1698 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
I0526 19:07:12.202960 1698 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0526 19:07:12.202999 1698 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 -j ACCEPT
I0526 19:07:12.204866 1698 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0526 19:07:12.205931 1698 iptables.go:167] Deleting iptables rule: -d 10.42.0.0/16 -j ACCEPT
I0526 19:07:12.207820 1698 iptables.go:167] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/24 -j RETURN
I0526 19:07:12.209118 1698 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 -j ACCEPT
I0526 19:07:12.210083 1698 iptables.go:167] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully
I0526 19:07:12.211653 1698 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
I0526 19:07:12.214128 1698 iptables.go:155] Adding iptables rule: -d 10.42.0.0/16 -j ACCEPT
I0526 19:07:12.217520 1698 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0526 19:07:12.225410 1698 iptables.go:155] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/24 -j RETURN
I0526 19:07:12.228914 1698 iptables.go:155] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully
部署一个演示程序
alias k='$(pwd)/k3s-arm64 kubectl'
k run hello-arm --image=registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1
k get pod
k logs -f hello-arm
输出下面日志确定OK
NAME READY STATUS RESTARTS AGE
hello-arm 0/1 Completed 0 2s
Hello, my architecture is Linux buildkitsandbox 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 aarch64 Linux
如何实现容器镜像多平台包构建 ？
呼之欲出buildx
啥子是buildx ? 自己bing, 文档地址：docker/buildx
安装
- buildx安装
# 从 https://github.com/docker/buildx/releases 下载 对应平台的buildx,比如： buildx-v0.4.1.darwin-amd64
mkdir -p $HOME/.docker/cli-plugins/
mv buildx-v0.4.1.darwin-amd64 $HOME/.docker/cli-plugins/docker-buildx
export DOCKER_CLI_EXPERIMENTAL=enabled
docker buildx version
输出下面日志是OK
github.com/docker/buildx v0.4.1 bda4882a65349ca359216b135896bddc1d92461c
- 创建builder的时候需要的镜像装备
docker pull registry.cn-hangzhou.aliyuncs.com/k8ops/moby-buildkit:buildx-stable-1
docker tag registry.cn-hangzhou.aliyuncs.com/k8ops/moby-buildkit:buildx-stable-1 moby/buildkit:buildx-stable-1
docker images |grep moby
输出效果如下：
moby/buildkit buildx-stable-1 f2a88cb62c92 5 weeks ago 82.8MB
registry.cn-hangzhou.aliyuncs.com/k8ops/moby-buildkit buildx-stable-1 f2a88cb62c92 5 weeks ago 82.8MB
- 创建builder
docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap
[+] Building 12.0s (1/1) FINISHED
=> [internal] booting buildkit 11.9s
=> => pulling image moby/buildkit:buildx-stable-1 10.9s
=> => creating container buildx_buildkit_mybuilder0 1.1s
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
创建hello镜像
file: hello.c
/*
* hello.c
*/
#include <stdio.h>
#include <stdlib.h>
#ifndef ARCH
#define ARCH "Undefined"
#endif
int main() {
printf("Hello, my architecture is %s\n", ARCH);
exit(0);
}
file: Dockerfile
#
# Dockerfile
#
FROM alpine AS builder
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories \
&& apk update
RUN apk add build-base
WORKDIR /home
COPY hello.c .
RUN gcc "-DARCH=\"`uname -a`\"" hello.c -o hello
FROM alpine
WORKDIR /home
COPY --from=builder /home/hello .
ENTRYPOINT ["./hello"]
mkdir huawei-arm-k3s
touch hello.c Dockerfile
docker buildx build --platform linux/arm,linux/arm64,linux/amd64 -t registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1 . --push
docker buildx imagetools inspect registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1
Name: registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest: sha256:6c7714c17223d788c0bf26f9f45e1af9c5edd3478c990c4051a72d8e9cd5aa5c
Manifests:
Name: registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1@sha256:9eb62f00457b9eff252fc6143dfd33d80d31a5013cf08a8ab0132394262198e7
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/arm64
Name: registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1@sha256:d7244652b1ce9b0bd1060be8a861a5aff2d5a893cd0ba3afb0db68a225034893
MediaType: application/vnd.docker.distribution.manifest.v2+json
Platform: linux/amd64
本地验证
docker run --rm -it registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1@sha256:d7244652b1ce9b0bd1060be8a861a5aff2d5a893cd0ba3afb0db68a225034893
Unable to find image 'registry.cn-hangzhou.aliyuncs.com/k8ops/hello:0.1@sha256:d7244652b1ce9b0bd1060be8a861a5aff2d5a893cd0ba3afb0db68a225034893' locally
sha256:d7244652b1ce9b0bd1060be8a861a5aff2d5a893cd0ba3afb0db68a225034893: Pulling from k8ops/hello
cbdbe7a5bc2a: Already exists
e26bd31f84ac: Already exists
b72c012d327f: Pull complete
Digest: sha256:d7244652b1ce9b0bd1060be8a861a5aff2d5a893cd0ba3afb0db68a225034893
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/k8ops/hello@sha256:d7244652b1ce9b0bd1060be8a861a5aff2d5a893cd0ba3afb0db68a225034893
Hello, my architecture is Linux buildkitsandbox 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 Linux
