Kubernetes编程—— 编写 Operator 的方案 —— 2、基于 sample-controller

编写 Operator 的方案 —— 2、基于 sample-controller

https://github.com/kubernetes/sample-controller

  首先我们将基于 k8s.io/sample-controller 来实现 cnat，通过直接使用 client-go 完成。

  sample-controller 使用 k8s.io/code-generator 来生成强类型的客户端、Informer、Lister 和深拷贝函数。如果你的自定义控制器中的 API 发生了变化，比如在自定义资源中添加了一个新的字段，你需要运行 update-codegen.sh 脚本来重新生成先前提到过的源代码文件。

  我们开始基于 sample-controller 使用 client-go 来实现我们的 cnat Operator 吧。

一、定义 CRD 资源

  1、首先，运行 go get k8s.io/sample-controller 获取相关的源代码及依赖并安装到系统中。

[root@JumperServer:zuoyang] # git clone https://github.com/kubernetes/sample-controller
Cloning into 'sample-controller'...
remote: Enumerating objects: 26102, done.
remote: Counting objects: 100% (1850/1850), done.
remote: Compressing objects: 100% (363/363), done.
remote: Total 26102 (delta 1564), reused 1751 (delta 1478), pack-reused 24252
Receiving objects: 100% (26102/26102), 11.64 MiB | 13.98 MiB/s, done.
Resolving deltas: 100% (18087/18087), done.
[root@JumperServer:zuoyang] # cd sample-controller/
[root@JumperServer:sample-controller] # ls -l
total 108
drwxr-xr-x 3 root root  4096 Aug 21 17:43 artifacts
-rw-r--r-- 1 root root   148 Aug 21 17:43 code-of-conduct.md
-rw-r--r-- 1 root root   756 Aug 21 17:43 CONTRIBUTING.md
-rw-r--r-- 1 root root 16506 Aug 21 17:43 controller.go
-rw-r--r-- 1 root root  9746 Aug 21 17:43 controller_test.go
drwxr-xr-x 3 root root  4096 Aug 21 17:43 docs
-rw-r--r-- 1 root root  2735 Aug 21 17:43 go.mod
-rw-r--r-- 1 root root 15455 Aug 21 17:43 go.sum
drwxr-xr-x 2 root root  4096 Aug 21 17:43 hack
-rw-r--r-- 1 root root 11358 Aug 21 17:43 LICENSE
-rw-r--r-- 1 root root  2946 Aug 21 17:43 main.go
-rw-r--r-- 1 root root   183 Aug 21 17:43 OWNERS
drwxr-xr-x 5 root root  4096 Aug 21 17:43 pkg
-rw-r--r-- 1 root root  7543 Aug 21 17:43 README.md
-rw-r--r-- 1 root root   550 Aug 21 17:43 SECURITY_CONTACTS
[root@JumperServer:sample-controller] # go get k8s.io/sample-controller
go: downloading k8s.io/api v0.0.0-20230810042731-2f6eec10c476
go: downloading k8s.io/apimachinery v0.0.0-20230815235016-14436eb53afd
go: downloading k8s.io/client-go v0.0.0-20230816000758-856e847bb7cb
[root@JumperServer:sample-controller] #　

  2、这个命令会启动自定义控制器，等待你注册 CRD 并创建一个自定义资源。

[root@JumperServer:sample-controller] # kubectl apply -f artifacts/examples/
crd-status-subresource.yaml  crd.yaml                     example-foo.yaml
[root@JumperServer:sample-controller] # kubectl apply -f artifacts/examples/crd.yaml
customresourcedefinition.apiextensions.k8s.io/foos.samplecontroller.k8s.io created
[root@JumperServer:sample-controller] #  cat artifacts/examples/crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: foos.samplecontroller.k8s.io
  # for more information on the below annotation, please see
  # https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2337-k8s.io-group-protection/README.md
  annotations:
    "api-approved.kubernetes.io": "unapproved, experimental-only; please get an approval from Kubernetes API reviewers if you're trying to develop a CRD in the *.k8s.io or *.kubernetes.io groups"
spec:
  group: samplecontroller.k8s.io
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        # schema used for validation
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                deploymentName:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
            status:
              type: object
              properties:
                availableReplicas:
                  type: integer
  names:
    kind: Foo
    plural: foos
  scope: Namespaced

3、如果成功注册了 CRD，运行下面的命令应该会看到相应的输出：

[root@JumperServer:sample-controller] # kubectl get crds
NAME                                             CREATED AT
applicationdrafts.orchestration.caicloud.io      2023-06-12T06:34:12Z
applications.orchestration.caicloud.io           2023-06-12T06:34:12Z
clusteroperations.operation.cce.io               2023-07-24T04:02:19Z
clusterquotas.tenant.caicloud.io                 2023-06-12T06:35:06Z
clusters.resource.caicloud.io                    2023-06-12T06:34:14Z
configclaims.config.caicloud.io                  2023-06-12T06:34:14Z
configreferences.config.caicloud.io              2023-06-12T06:34:14Z
configs.resource.caicloud.io                     2023-06-12T06:34:14Z
crontabs.zuoyang.tech                            2023-07-10T10:49:55Z
drainages.node.cce.io                            2023-07-24T04:58:53Z
extendedresources.resource.caicloud.io           2023-06-12T06:34:14Z
foos.samplecontroller.k8s.io                     2023-08-21T09:49:51Z
geips.geip.cce.io                                2023-06-12T06:27:48Z
infranetworks.resource.caicloud.io               2023-06-12T06:34:14Z
loadbalancers.loadbalance.caicloud.io            2023-06-12T06:34:25Z
machineautoscalinggroups.resource.caicloud.io    2023-06-12T06:34:14Z
machines.resource.caicloud.io                    2023-06-12T06:34:14Z
network-attachment-definitions.k8s.cni.cncf.io   2023-06-12T06:28:04Z
networkpolicies.networking.caicloud.io           2023-06-12T06:35:07Z
networks.resource.caicloud.io                    2023-06-12T06:32:47Z
nodeclaims.resource.caicloud.io                  2023-06-12T06:34:14Z
nodeconfigs.config.k8s.io                        2023-06-12T06:30:52Z
nodelocalstorages.resource.caicloud.io           2023-06-12T06:34:39Z
nodelocalvolumes.localvolume.everest.io          2023-06-12T06:28:24Z
nodeoperations.operation.cce.io                  2023-07-24T04:02:20Z
packageversions.version.cce.io                   2023-06-12T06:28:15Z
partitions.tenant.caicloud.io                    2023-06-12T06:35:06Z
permissions.rbac.cce.io                          2023-06-12T06:28:13Z
podnetworkinterfaceclaims.crd.yangtse.cni        2023-06-12T06:28:03Z
podnetworkinterfacepools.crd.yangtse.cni         2023-06-12T06:28:03Z
podnetworkinterfaceqosconfigs.crd.yangtse.cni    2023-06-12T06:28:03Z
podnetworkinterfaces.crd.yangtse.cni             2023-06-12T06:28:03Z
releasehistories.release.caicloud.io             2023-06-12T06:33:15Z
releases.release.caicloud.io                     2023-06-12T06:32:59Z
requirementgaps.resource.caicloud.io             2023-06-12T06:34:14Z
resourceclasses.resource.caicloud.io             2023-06-12T06:34:14Z
snapshots.resource.caicloud.io                   2023-06-12T06:35:01Z
springclouds.microservice.caicloud.io            2023-06-12T06:34:52Z
tags.resource.caicloud.io                        2023-06-12T06:34:14Z
tenants.tenant.caicloud.io                       2023-06-12T06:35:05Z
volumesnapshotclasses.snapshot.storage.k8s.io    2023-06-12T06:32:14Z
volumesnapshotcontents.snapshot.storage.k8s.io   2023-06-12T06:32:14Z
volumesnapshots.snapshot.storage.k8s.io          2023-06-12T06:32:14Z
workloads.workload.caicloud.io                   2023-06-12T06:34:53Z

　　4、下一步，创建一个名为 foo.samplecontroller.k8s.io/example-foo 的自定义资源，并确认控制器有没有正常工作：

[root@JumperServer:sample-controller] # kubectl apply -f artifacts/examples/example-foo.yaml
foo.samplecontroller.k8s.io/example-foo created
[root@JumperServer:sample-controller] # cat artifacts/examples/example-foo.yaml
apiVersion: samplecontroller.k8s.io/v1alpha1
kind: Foo
metadata:
  name: example-foo
spec:
  deploymentName: example-foo
  replicas: 1
[root@JumperServer:sample-controller] # kubectl get po,rs,deploy,foo
NAME                                      AGE
foo.samplecontroller.k8s.io/example-foo   46s

 二、编写 CRD 业务逻辑

　在开始业务逻辑之前，让我们先把现有的 sample-controller 复制一个新的 cnat：

[root@JumperServer:sample-controller] # cp sample-controller cnat
[root@JumperServer:sample-controller] # ll
total 8
drwxr-xr-x 8 root root 4096 Aug 21 17:57 cnat
drwxr-xr-x 8 root root 4096 Aug 21 17:49 sample-controller
[root@JumperServer:sample-controller] # cd cnat/artifacts/examples/
[root@JumperServer:sample-controller] # ll
total 12
-rw-r--r-- 1 root root 1315 Aug 21 17:49 crd-status-subresource.yaml
-rw-r--r-- 1 root root 1190 Aug 21 17:49 crd.yaml
-rw-r--r-- 1 root root  135 Aug 21 17:49 example-foo.yaml
[root@JumperServer:sample-controller] # cp crd.yaml cnat-crd.yaml
[root@JumperServer:sample-controller] # cp example-foo.yaml cnat-example.yaml
[root@JumperServer:sample-controller] # cat cnat-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ats.cnat.zuoyang.tech
  # for more information on the below annotation, please see
  # https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2337-k8s.io-group-protection/README.md
  annotations:
    "api-approved.kubernetes.io": "unapproved, experimental-only; please get an approval from Kubernetes API reviewers if you're trying to develop a CRD in the *.k8s.io or *.kubernetes.io groups"
spec:
  group: cnat.zuoyang.tech
  versions:
    - name: v1alpha1
      kind: Foo
      plural: foos
  scope: Namespaced
[root@JumperServer:sample-controller] # cat cnat-example.yaml
apiVersion: cnat.zuoyang.tech/v1alpha1
kind: At
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: example-at
spec:
  schedule: "2019-04-12T10:12:00Z"
  command: "echo YAY"

注意：只要 API 类型的定义发生了变化，比如你向 CRD 中添加了一个 At 字段，你就需要重新运行 update-codegen.sh 脚本：

[root@JumperServer:sample-controller] # cat hack/update-codegen.sh
#!/usr/bin/env bash

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -o errexit
set -o nounset
set -o pipefail

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}

source "${CODEGEN_PKG}/kube_codegen.sh"

# generate the code with:
# --output-base    because this script should also be able to run inside the vendor dir of
#                  k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
#                  instead of the $GOPATH directly. For normal projects this can be dropped.

kube::codegen::gen_helpers \
    --input-pkg-root k8s.io/sample-controller/pkg/apis \
    --output-base "$(dirname "${BASH_SOURCE[0]}")/../../.." \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt"

kube::codegen::gen_client \
    --with-watch \
    --input-pkg-root k8s.io/sample-controller/pkg/apis \
    --output-pkg-root k8s.io/sample-controller/pkg/generated \
    --output-base "$(dirname "${BASH_SOURCE[0]}")/../../.." \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt"

这里有个问题，我执行这个脚本报错：

./hack/verify-codegen.sh
/zuoyang/cnat/hack/update-codegen.sh: line 24: ../code-generator/kube_codegen.sh: No such file or directory

故障排查后，找到解决办法：

[root@JumperServer:sample-controller] # ll
total 8
drwxr-xr-x 8 root root 4096 Aug 21 17:57 cnat
drwxr-xr-x 8 root root 4096 Aug 21 17:49 sample-controller
[root@JumperServer:sample-controller] #  git clone https://github.com/kubernetes/code-generator.git
Cloning into 'code-generator'...
remote: Enumerating objects: 13112, done.
remote: Counting objects: 100% (3548/3548), done.
remote: Compressing objects: 100% (627/627), done.
remote: Total 13112 (delta 2955), reused 3405 (delta 2865), pack-reused 9564
Receiving objects: 100% (13112/13112), 9.78 MiB | 6.19 MiB/s, done.
Resolving deltas: 100% (7683/7683), done.
[root@cce-poc-daxs1 zuoyang]# ll
total 12
drwxr-xr-x 8 root root 4096 Aug 21 17:57 cnat
drwxr-xr-x 8 root root 4096 Aug 21 18:21 code-generator
drwxr-xr-x 8 root root 4096 Aug 21 17:49 sample-controller
[root@cce-poc-daxs1 zuoyang]# cd cnat/
[root@JumperServer:sample-controller] # ls -l
total 108
drwxr-xr-x 3 root root  4096 Aug 21 17:57 artifacts
-rw-r--r-- 1 root root   148 Aug 21 17:57 code-of-conduct.md
-rw-r--r-- 1 root root   756 Aug 21 17:57 CONTRIBUTING.md
-rw-r--r-- 1 root root 16506 Aug 21 17:57 controller.go
-rw-r--r-- 1 root root  9746 Aug 21 17:57 controller_test.go
drwxr-xr-x 3 root root  4096 Aug 21 17:57 docs
-rw-r--r-- 1 root root  2735 Aug 21 17:57 go.mod
-rw-r--r-- 1 root root 15455 Aug 21 17:57 go.sum
drwxr-xr-x 2 root root  4096 Aug 21 18:16 hack
-rw-r--r-- 1 root root 11358 Aug 21 17:57 LICENSE
-rw-r--r-- 1 root root  2946 Aug 21 17:57 main.go
-rw-r--r-- 1 root root   183 Aug 21 17:57 OWNERS
drwxr-xr-x 5 root root  4096 Aug 21 17:57 pkg
-rw-r--r-- 1 root root  7543 Aug 21 17:57 README.md
-rw-r--r-- 1 root root   550 Aug 21 17:57 SECURITY_CONTACTS
[root@JumperServer:sample-controller] # ./hack/
update-codegen.sh  verify-codegen.sh
[root@JumperServer:sample-controller] # ./hack/update-codegen.sh
go: downloading k8s.io/gengo v0.0.0-20220902162205-c0856e24416d
go: downloading k8s.io/klog/v2 v2.100.1
go: downloading golang.org/x/tools v0.8.0
go: downloading github.com/go-logr/logr v1.2.4
go: downloading golang.org/x/mod v0.10.0
go: downloading golang.org/x/sys v0.10.0
fatal: ambiguous argument ':(glob)/k8s.io/sample-controller/pkg/apis/**/*.go': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument ':(glob)/k8s.io/sample-controller/pkg/apis/**/*.go': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument ':(glob)/k8s.io/sample-controller/pkg/apis/**/*.go': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
go: downloading github.com/google/gnostic-models v0.6.8
go: downloading k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9
go: downloading google.golang.org/protobuf v1.30.0
go: downloading github.com/go-openapi/jsonreference v0.20.2
go: downloading github.com/google/gofuzz v1.2.0
fatal: ambiguous argument ':(glob)/k8s.io/sample-controller/pkg/apis/**/types.go': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'