loongarch64架构下Kubernetes组件编译

1、编译环境

• OS ：Loongnix-Server Linux release 8.3
• CPU指令集 ： loongarch64
• 平台 ： 龙芯
• go版本 : go version go1.15.6 linux/loong64

2、go和docker安装

docker安装：

?

1	yum -y install docker-ce

golang安装：

?

1	yum -y install golang

默认安装golang版本是 go version go1.15.6 linux/loong64，如果需要升级golang版本的话可以

?

1 2	yum -y update yum search golang

然后根据search结果yum -y install指定golang版本。

3、编译kubernetes源码

kubernetes全部组件编译：

?

1	KUBE_BUILD_PLATFORMS=linux/loong64 make all

编译指定kubernetes组件：

?

1	KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet GOFLAGS=-v GOGCFLAGS="-N -l"

编译完成后，可以在 _output/local/bin/linux/loong64/中找到需要的二进制文件。

1）下载指定版本kubernetes源码

从github获取指定版本的k8s源码

?

1	wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.21.5.tar.gz

解压并进入到kubernetes-1.21.5目录

?

1	tar -zxvf v1.21.5.tar.gz && cd kubernetes-1.21.5

注意：本文以kubernetes1.21.5版本源码编译为例，其他kubernetes版本编译步骤和kubernetes1.21.5版本源码编译步骤一致。

2）组件编译及错误处理

2.1） 编译kubelet

?

1	KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet

编译发现报如下错误：

?

1 2	[root@master1 kubernetes-1.21.5]#KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet -bash: make: command not found

安装一下make即可：

?

1	yum -y install make

安装完make后重新编译kubelet:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet !!! [0509 06:14:35] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.   Detected go version: go version go1.15.6 linux/loong64. Kubernetes requires go1.16.0 or greater. Please install go1.16.0 or later.   !!! [0509 06:14:35] Call tree: !!! [0509 06:14:35]  1: hack/run-in-gopath.sh:31 kube::golang::setup_env(...) !!! [0509 06:14:35] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.   Detected go version: go version go1.15.6 linux/loong64. Kubernetes requires go1.16.0 or greater. Please install go1.16.0 or later.   !!! [0509 06:14:35] Call tree: !!! [0509 06:14:35]  1: /root/go/kubernetes-1.21.5/hack/lib/golang.sh:786 kube::golang::setup_env(...) !!! [0509 06:14:35]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! [0509 06:14:35] Call tree: !!! [0509 06:14:35]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) make[1]: *** [Makefile.generated_files:209: _output/bin/prerelease-lifecycle-gen] Error 1 make: *** [Makefile:551: generated_files] Error 2

首先会发现golang版本太低，这个直接yum search golang然后通过yum安装>=1.16.0以上版本的即可，详细步骤不再赘余,升级完golang版本后重新编译kubeadm:

?

1 2 3 4 5

[root@master1 kubernetes-1.21.5]# go version go version go1.18 linux/loong64 [root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet !!! [0509 06:19:40] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le. !!! [0509 06:19:45] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.

会发现kubernetes源码不支持loongarch64指令集问题，下面主要说一下如何解决kubernetes源码不支持loongarch64指令集问题。根据安装完make后重新编译kubelet错误提示，编辑hack/run-in-gopath.sh,发现其引用了hack/lib/init.sh，然后继续编辑hack/lib/init.sh，并递归检查他们依赖的.sh文件，查看这些文件中是否存在CPU指令集相关的逻辑判断，如果存在的话加上龙芯平台loongarch64架构的指令集，这里直接说下哪些文件需要调整：

（1）修改hack/lib/golang.sh文件

在KUBE_SUPPORTED_SERVER_PLATFORMS ， KUBE_SUPPORTED_NODE_PLATFORMS ，KUBE_SUPPORTED_CLIENT_PLATFORMS，KUBE_SUPPORTED_TEST_PLATFORMS分别添加上linux/loong64

（2）修改hack/lib/util.sh文件

在kube::util::host_arch() 里添加 

?

1 2 3

loongarch64*)   host_arch=loongarch64   ;;

（3）修改hack/local-up-cluster.sh文件

?

1 2 3

loongarch64*)   host_arch=loongarch64   ;;

（4）修改vendor/github.com/opencontainers/runc/libcontainer/system/syscall_linux_64.go文件

第二行注释后面追加loong64

?

1	//+build arm64 amd64 mips mipsle mips64 mips64le ppc ppc64 ppc64le riscv64 s390x loong64

修改完以上4个文件后再次编译kubelet

?

1 2 3 4 5 6 7 8 9

root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet +++ [0509 06:58:10] Building go targets for linux/loong64:     ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen find: 'rsync': No such file or directory find: 'rsync': No such file or directory Generating prerelease lifecycle code for 27 targets ./hack/run-in-gopath.sh: line 34: _output/bin/prerelease-lifecycle-gen: Permission denied make[1]: *** [Makefile.generated_files:160: gen_prerelease_lifecycle] Error 1 make: *** [Makefile:551: generated_files] Error 2

发现找不到rsync，需要安装rsync并赋予prerelease-lifecycle-gen可执行权限

?

1 2	yum -y install rsync chmod +x _output/bin/prerelease-lifecycle-gen

解决完上面错误后再次编译kubelet

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet Generating prerelease lifecycle code for 27 targets +++ [0509 07:04:53] Building go targets for linux/loong64:     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen Generating deepcopy code for 227 targets +++ [0509 07:05:21] Building go targets for linux/loong64:     ./vendor/k8s.io/code-generator/cmd/defaulter-gen Generating defaulter code for 89 targets +++ [0509 07:05:57] Building go targets for linux/loong64:     ./vendor/k8s.io/code-generator/cmd/conversion-gen Generating conversion code for 124 targets +++ [0509 07:07:12] Building go targets for linux/loong64:     ./vendor/k8s.io/kube-openapi/cmd/openapi-gen Generating openapi code for KUBE Generating openapi code for AGGREGATOR Generating openapi code for APIEXTENSIONS Generating openapi code for CODEGEN Generating openapi code for SAMPLEAPISERVER +++ [0509 07:08:08] Building go targets for linux/loong64:     ./vendor/github.com/go-bindata/go-bindata/go-bindata +++ [0509 07:08:14] Building go targets for linux/loong64:     cmd/kubelet # k8s.io/kubernetes/vendor/golang.org/x/sys/unix vendor/golang.org/x/sys/unix/affinity_linux.go:14:35: undefined: _NCPUBITS vendor/golang.org/x/sys/unix/affinity_linux.go:17:25: undefined: cpuMask vendor/golang.org/x/sys/unix/affinity_linux.go:50:27: undefined: cpuMask vendor/golang.org/x/sys/unix/sockcmsg_unix.go:34:9: undefined: Cmsghdr vendor/golang.org/x/sys/unix/ztypes_linux.go:23:11: undefined: Timespec vendor/golang.org/x/sys/unix/ztypes_linux.go:24:11: undefined: Timespec vendor/golang.org/x/sys/unix/ztypes_linux.go:1132:12: undefined: SockaddrStorage vendor/golang.org/x/sys/unix/ztypes_linux.go:2313:8: undefined: Timespec vendor/golang.org/x/sys/unix/ztypes_linux.go:2806:8: undefined: TIPCSubscr vendor/golang.org/x/sys/unix/ztypes_linux.go:3214:10: undefined: PPSKInfo vendor/golang.org/x/sys/unix/affinity_linux.go:50:27: too many errors

发现报golang.org/x/sys/unix包下某些函数没有定义等错误，这是因为它找不到当前平台所对应的硬件信息文件。解决办法是，用GOROOT下的golang.org/x/sys/unix包替换vendor/golang.org/x/sys/路径下面的unix包。

?

1	sudo cp -r /usr/lib/golang-1.18/src/cmd/vendor/golang.org/x/sys/unix vendor/golang.org/x/sys/

替换完unix包后再次重新编译kubelet：

?

1 2 3 4

[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet +++ [0509 07:35:07] Building go targets for linux/loong64:     cmd/kubelet [root@master1 kubernetes-1.21.5]#

经过几分钟等待发现kubelet编译成功。

这里可以到_output/bin目录下验证下kubelet是否编译成功。

?

1 2	[root@master1 bin]# ./kubelet --version Kubernetes v1.21.5

可以看到kebelet组件没问题，至此kubernetes kueblet组件编译成功。

2.2） 编译kubernetes其他组件

编译kubernetes其他组件（如kubeadm、kubectl...）步骤和编译kubelet一致，这里不再赘余。