loongarch64架构下Kubernetes组件编译

1、编译环境

OS ：Loongnix-Server Linux release 8.3
CPU指令集： loongarch64
平台：龙芯
go版本 : go version go1.15.6 linux/loong64

2、go和docker安装

docker安装：

yum -y install docker-ce

golang安装：

yum -y install golang

默认安装golang版本是 go version go1.15.6 linux/loong64，如果需要升级golang版本的话可以

yum -y update
yum search golang

然后根据search结果yum -y install指定golang版本。

3、编译kubernetes源码

kubernetes全部组件编译：

KUBE_BUILD_PLATFORMS=linux/loong64 make all

编译指定kubernetes组件：

KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet GOFLAGS=-v GOGCFLAGS="-N -l"

编译完成后，可以在 _output/local/bin/linux/loong64/中找到需要的二进制文件。

1）下载指定版本kubernetes源码

从github获取指定版本的k8s源码

wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.21.5.tar.gz

解压并进入到kubernetes-1.21.5目录

tar -zxvf v1.21.5.tar.gz && cd kubernetes-1.21.5

注意：本文以kubernetes1.21.5版本源码编译为例，其他kubernetes版本编译步骤和kubernetes1.21.5版本源码编译步骤一致。

2）组件编译及错误处理

2.1）编译kubelet

KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet

编译发现报如下错误：

[root@master1 kubernetes-1.21.5]#KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet
-bash: make: command not found

安装一下make即可：

 yum -y install make

安装完make后重新编译kubelet:

[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet
!!! [0509 06:14:35] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.

Detected go version: go version go1.15.6 linux/loong64.
Kubernetes requires go1.16.0 or greater.
Please install go1.16.0 or later.

!!! [0509 06:14:35] Call tree:
!!! [0509 06:14:35]  1: hack/run-in-gopath.sh:31 kube::golang::setup_env(...)
!!! [0509 06:14:35] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.

Detected go version: go version go1.15.6 linux/loong64.
Kubernetes requires go1.16.0 or greater.
Please install go1.16.0 or later.

!!! [0509 06:14:35] Call tree:
!!! [0509 06:14:35]  1: /root/go/kubernetes-1.21.5/hack/lib/golang.sh:786 kube::golang::setup_env(...)
!!! [0509 06:14:35]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0509 06:14:35] Call tree:
!!! [0509 06:14:35]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make[1]: *** [Makefile.generated_files:209: _output/bin/prerelease-lifecycle-gen] Error 1
make: *** [Makefile:551: generated_files] Error 2

首先会发现golang版本太低，这个直接yum search golang然后通过yum安装>=1.16.0以上版本的即可，详细步骤不再赘余,升级完golang版本后重新编译kubeadm:

[root@master1 kubernetes-1.21.5]# go version
go version go1.18 linux/loong64
[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet
!!! [0509 06:19:40] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.
!!! [0509 06:19:45] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.

会发现kubernetes源码不支持loongarch64指令集问题，下面主要说一下如何解决kubernetes源码不支持loongarch64指令集问题。根据安装完make后重新编译kubelet错误提示，编辑hack/run-in-gopath.sh,发现其引用了hack/lib/init.sh，然后继续编辑hack/lib/init.sh，并递归检查他们依赖的.sh文件，查看这些文件中是否存在CPU指令集相关的逻辑判断，如果存在的话加上龙芯平台loongarch64架构的指令集，这里直接说下哪些文件需要调整：

（1）修改hack/lib/golang.sh文件

在KUBE_SUPPORTED_SERVER_PLATFORMS ， KUBE_SUPPORTED_NODE_PLATFORMS ，KUBE_SUPPORTED_CLIENT_PLATFORMS，KUBE_SUPPORTED_TEST_PLATFORMS分别添加上linux/loong64

（2）修改hack/lib/util.sh文件

在kube::util::host_arch() 里添加

loongarch64*)
  host_arch=loongarch64
  ;;

（3）修改hack/local-up-cluster.sh文件

loongarch64*)
  host_arch=loongarch64
  ;;

（4）修改vendor/github.com/opencontainers/runc/libcontainer/system/syscall_linux_64.go文件

第二行注释后面追加loong64

//+build arm64 amd64 mips mipsle mips64 mips64le ppc ppc64 ppc64le riscv64 s390x loong64

修改完以上4个文件后再次编译kubelet

root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet
+++ [0509 06:58:10] Building go targets for linux/loong64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
find: 'rsync': No such file or directory
find: 'rsync': No such file or directory
Generating prerelease lifecycle code for 27 targets
./hack/run-in-gopath.sh: line 34: _output/bin/prerelease-lifecycle-gen: Permission denied
make[1]: *** [Makefile.generated_files:160: gen_prerelease_lifecycle] Error 1
make: *** [Makefile:551: generated_files] Error 2

发现找不到rsync，需要安装rsync并赋予prerelease-lifecycle-gen可执行权限

 yum -y install rsync
 chmod +x _output/bin/prerelease-lifecycle-gen

解决完上面错误后再次编译kubelet

[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet
Generating prerelease lifecycle code for 27 targets
+++ [0509 07:04:53] Building go targets for linux/loong64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 227 targets
+++ [0509 07:05:21] Building go targets for linux/loong64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 89 targets
+++ [0509 07:05:57] Building go targets for linux/loong64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 124 targets
+++ [0509 07:07:12] Building go targets for linux/loong64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0509 07:08:08] Building go targets for linux/loong64:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0509 07:08:14] Building go targets for linux/loong64:
    cmd/kubelet
# k8s.io/kubernetes/vendor/golang.org/x/sys/unix
vendor/golang.org/x/sys/unix/affinity_linux.go:14:35: undefined: _NCPUBITS
vendor/golang.org/x/sys/unix/affinity_linux.go:17:25: undefined: cpuMask
vendor/golang.org/x/sys/unix/affinity_linux.go:50:27: undefined: cpuMask
vendor/golang.org/x/sys/unix/sockcmsg_unix.go:34:9: undefined: Cmsghdr
vendor/golang.org/x/sys/unix/ztypes_linux.go:23:11: undefined: Timespec
vendor/golang.org/x/sys/unix/ztypes_linux.go:24:11: undefined: Timespec
vendor/golang.org/x/sys/unix/ztypes_linux.go:1132:12: undefined: SockaddrStorage
vendor/golang.org/x/sys/unix/ztypes_linux.go:2313:8: undefined: Timespec
vendor/golang.org/x/sys/unix/ztypes_linux.go:2806:8: undefined: TIPCSubscr
vendor/golang.org/x/sys/unix/ztypes_linux.go:3214:10: undefined: PPSKInfo
vendor/golang.org/x/sys/unix/affinity_linux.go:50:27: too many errors

发现报golang.org/x/sys/unix包下某些函数没有定义等错误，这是因为它找不到当前平台所对应的硬件信息文件。解决办法是，用GOROOT下的golang.org/x/sys/unix包替换vendor/golang.org/x/sys/路径下面的unix包。

sudo cp -r /usr/lib/golang-1.18/src/cmd/vendor/golang.org/x/sys/unix vendor/golang.org/x/sys/

替换完unix包后再次重新编译kubelet：

[root@master1 kubernetes-1.21.5]# KUBE_BUILD_PLATFORMS=linux/loong64 make all WHAT=cmd/kubelet
+++ [0509 07:35:07] Building go targets for linux/loong64:
    cmd/kubelet
[root@master1 kubernetes-1.21.5]#

经过几分钟等待发现kubelet编译成功。

这里可以到_output/bin目录下验证下kubelet是否编译成功。

[root@master1 bin]# ./kubelet --version
Kubernetes v1.21.5

可以看到kebelet组件没问题，至此kubernetes kueblet组件编译成功。

2.2）编译kubernetes其他组件

编译kubernetes其他组件（如kubeadm、kubectl...）步骤和编译kubelet一致，这里不再赘余。

posted @ 2022-05-09 15:56 人艰不拆_zmc 阅读(2402) 评论(0) 编辑收藏举报

刷新页面返回顶部