Kubebuilder实践

1、常见的开发框架

2、kubebuilder安装

kubebuilder book:https://book.kubebuilder.io/quick-start.html

kubebuilder GitHub:https://github.com/kubernetes-sigs/kubebuilder

kubebuilder安装比较的简单。由于对于windows支持不是很好,建议在Linux上安装。

必要条件:

Prerequisites

  • go version v1.15+ (kubebuilder v3.0 < v3.1).
  • go version v1.16+ (kubebuilder v3.1 < v3.3).
  • go version v1.17+ (kubebuilder v3.3+).
  • docker version 17.03+.
  • kubectl version v1.11.3+.
  • Access to a Kubernetes v1.11.3+ cluster.

查看go 、docker和kubernetes版本

[root@master ~]# go version
go version go1.17.7 linux/amd64
[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:10:45Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5", GitCommit:"aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:"clean", BuildDate:"2021-09-15T21:04:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:44 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[root@master ~]# 

注:

(1)关于在centos8.2中安装gov1.17,见:centos 8.2中安装go 1.17

由于我们的go 版本为1.17,所以应该下载kubebuilder版本v3.3+

下载并安装kuberbuilder

# download kubebuilder and install locally.
$ curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
$ chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

3、kubebuilder初步实践

初始化

[root@master ~]# mkdir -p $GOPATH/src/kubebuilder-demo
[root@master ~]# cd $GOPATH/src/kubebuilder-demo
[root@master kubebuilder-demo]# kubebuilder init --domain demo.kubebuilder.io
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.11.0
Update dependencies:
$ go mod tidy
go: downloading github.com/stretchr/testify v1.7.0
go: downloading github.com/onsi/gomega v1.17.0
go: downloading github.com/onsi/ginkgo v1.16.5
go: downloading github.com/go-logr/zapr v1.2.0
go: downloading go.uber.org/zap v1.19.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest v14.2.0+incompatible
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading go.uber.org/goleak v1.1.12
go: downloading go.uber.org/atomic v1.7.0
go: downloading go.uber.org/multierr v1.6.0
go: downloading github.com/benbjohnson/clock v1.1.0
go: downloading gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading github.com/Azure/go-autorest/tracing v0.6.0
go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.1
go: downloading cloud.google.com/go v0.81.0
go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0
go: downloading github.com/form3tech-oss/jwt-go v3.2.3+incompatible
go: downloading golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: downloading github.com/nxadm/tail v1.4.8
go: downloading github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: downloading github.com/kr/text v0.2.0
Next: define a resource with:

注意:如果你不是在$GOPATH/src路径下进行初始化,请添加上--repo参数作为项目MODULE的名称

mkdir -p ~/projects/guestbook
cd ~/projects/guestbook
kubebuilder init --domain my.domain --repo my.domain/guestbook

Developing in $GOPATH

If your project is initialized within GOPATH, the implicitly called go mod init will interpolate the module path for you. Otherwise --repo= must be set.

Read the Go modules blogpost if unfamiliar with the module system.

创建API

[root@master kubebuilder-demo]# kubebuilder create api --group myapp --version v1 --kind Redis
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/redis_types.go
controllers/redis_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.8.0
go: downloading sigs.k8s.io/controller-tools v0.8.0
go: downloading github.com/spf13/cobra v1.2.1
go: downloading github.com/gobuffalo/flect v0.2.3
go: downloading golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go: downloading github.com/fatih/color v1.12.0
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/google/go-cmp v0.5.6
go: downloading golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go: downloading golang.org/x/mod v0.4.2
go get: installing executables with 'go get' in module mode is deprecated.
        To adjust and download dependencies of the current module, use 'go get -d'.
        To install using requirements of the current module, use 'go install'.
        To install ignoring the current module, use 'go install' with a version,
        like 'go install example.com/cmd@latest'.
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.12.0
go get: added github.com/go-logr/logr v1.2.0
go get: added github.com/gobuffalo/flect v0.2.3
go get: added github.com/gogo/protobuf v1.3.2
go get: added github.com/google/go-cmp v0.5.6
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.12
go get: added github.com/mattn/go-colorable v0.1.8
go get: added github.com/mattn/go-isatty v0.0.12
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.2
go get: added github.com/spf13/cobra v1.2.1
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.4.2
go get: added golang.org/x/net v0.0.0-20210825183410-e898025ed96a
go get: added golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go get: added golang.org/x/text v0.3.7
go get: added golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.4.0
go get: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go get: added k8s.io/api v0.23.0
go get: added k8s.io/apiextensions-apiserver v0.23.0
go get: added k8s.io/apimachinery v0.23.0
go get: added k8s.io/klog/v2 v2.30.0
go get: added k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b
go get: added sigs.k8s.io/controller-tools v0.8.0
go get: added sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
go get: added sigs.k8s.io/structured-merge-diff/v4 v4.1.2
go get: added sigs.k8s.io/yaml v1.3.0
/root/gopath/src/kubebuilder-demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
[root@master kubebuilder-demo]# 

make install

[root@master kubebuilder-demo]# make install
/root/gopath/src/kubebuilder-demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go: downloading sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go: downloading k8s.io/client-go v0.18.10
go: downloading github.com/spf13/cobra v1.0.0
go: downloading sigs.k8s.io/kustomize/api v0.6.5
go: downloading sigs.k8s.io/kustomize/cmd/config v0.8.5
go: downloading github.com/evanphx/json-patch v4.9.0+incompatible
go: downloading k8s.io/apimachinery v0.18.10
go: downloading sigs.k8s.io/kustomize/kyaml v0.9.4
go: downloading github.com/gogo/protobuf v1.3.1
go: downloading k8s.io/klog v1.0.0
go: downloading sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go: downloading sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e
go: downloading k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go: downloading k8s.io/api v0.18.10
go: downloading github.com/olekukonko/tablewriter v0.0.4
go: downloading github.com/go-errors/errors v1.0.1
go: downloading gopkg.in/yaml.v2 v2.3.0
go: downloading github.com/yujunz/go-getter v1.4.1-lite
go: downloading github.com/json-iterator/go v1.1.8
go: downloading github.com/googleapis/gnostic v0.1.0
go: downloading github.com/Azure/go-autorest/autorest v0.9.0
go: downloading github.com/Azure/go-autorest/autorest/adal v0.5.0
go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go: downloading github.com/gophercloud/gophercloud v0.1.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go: downloading github.com/stretchr/testify v1.6.1
go: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go: downloading github.com/go-openapi/strfmt v0.19.5
go: downloading github.com/go-openapi/validate v0.19.8
go: downloading github.com/mattn/go-runewidth v0.0.7
go: downloading github.com/hashicorp/go-multierror v1.1.0
go: downloading golang.org/x/net v0.0.0-20200625001655-4c5254603344
go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go: downloading github.com/hashicorp/go-cleanhttp v0.5.0
go: downloading github.com/hashicorp/go-safetemp v1.0.0
go: downloading github.com/hashicorp/go-version v1.1.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/mitchellh/go-testing-interface v1.0.0
go: downloading github.com/ulikunitz/xz v0.5.5
go: downloading github.com/Azure/go-autorest/logger v0.1.0
go: downloading github.com/Azure/go-autorest/tracing v0.5.0
go: downloading github.com/Azure/go-autorest/autorest/date v0.1.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading cloud.google.com/go v0.38.0
go: downloading google.golang.org/appengine v1.5.0
go: downloading github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go: downloading github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go: downloading github.com/go-openapi/errors v0.19.2
go: downloading github.com/mitchellh/mapstructure v1.1.2
go: downloading go.mongodb.org/mongo-driver v1.1.2
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/go-openapi/analysis v0.19.5
go: downloading github.com/go-openapi/loads v0.19.4
go: downloading github.com/go-openapi/runtime v0.19.4
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: downloading k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go: downloading golang.org/x/text v0.3.2
go: downloading github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go: downloading golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go: downloading github.com/go-stack/stack v1.8.0
go get: installing executables with 'go get' in module mode is deprecated.
        To adjust and download dependencies of the current module, use 'go get -d'.
        To install using requirements of the current module, use 'go install'.
        To install ignoring the current module, use 'go install' with a version,
        like 'go install example.com/cmd@latest'.
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.
go get: added cloud.google.com/go v0.38.0
go get: added github.com/Azure/go-autorest/autorest v0.9.0
go get: added github.com/Azure/go-autorest/autorest/adal v0.5.0
go get: added github.com/Azure/go-autorest/autorest/date v0.1.0
go get: added github.com/Azure/go-autorest/logger v0.1.0
go get: added github.com/Azure/go-autorest/tracing v0.5.0
go get: added github.com/PuerkitoBio/purell v1.1.1
go get: added github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go get: added github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go get: added github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go get: added github.com/davecgh/go-spew v1.1.1
go get: added github.com/dgrijalva/jwt-go v3.2.0+incompatible
go get: added github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go get: added github.com/evanphx/json-patch v4.9.0+incompatible
go get: added github.com/go-errors/errors v1.0.1
go get: added github.com/go-openapi/analysis v0.19.5
go get: added github.com/go-openapi/errors v0.19.2
go get: added github.com/go-openapi/jsonpointer v0.19.3
go get: added github.com/go-openapi/jsonreference v0.19.3
go get: added github.com/go-openapi/loads v0.19.4
go get: added github.com/go-openapi/runtime v0.19.4
go get: added github.com/go-openapi/spec v0.19.5
go get: added github.com/go-openapi/strfmt v0.19.5
go get: added github.com/go-openapi/swag v0.19.5
go get: added github.com/go-openapi/validate v0.19.8
go get: added github.com/go-stack/stack v1.8.0
go get: added github.com/gogo/protobuf v1.3.1
go get: added github.com/golang/protobuf v1.3.2
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go get: added github.com/googleapis/gnostic v0.1.0
go get: added github.com/gophercloud/gophercloud v0.1.0
go get: added github.com/hashicorp/errwrap v1.0.0
go get: added github.com/hashicorp/go-cleanhttp v0.5.0
go get: added github.com/hashicorp/go-multierror v1.1.0
go get: added github.com/hashicorp/go-safetemp v1.0.0
go get: added github.com/hashicorp/go-version v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.8
go get: added github.com/mailru/easyjson v0.7.0
go get: added github.com/mattn/go-runewidth v0.0.7
go get: added github.com/mitchellh/go-homedir v1.1.0
go get: added github.com/mitchellh/go-testing-interface v1.0.0
go get: added github.com/mitchellh/mapstructure v1.1.2
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go get: added github.com/olekukonko/tablewriter v0.0.4
go get: added github.com/pkg/errors v0.9.1
go get: added github.com/pmezard/go-difflib v1.0.0
go get: added github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go get: added github.com/spf13/cobra v1.0.0
go get: added github.com/spf13/pflag v1.0.5
go get: added github.com/stretchr/testify v1.6.1
go get: added github.com/ulikunitz/xz v0.5.5
go get: added github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go get: added github.com/yujunz/go-getter v1.4.1-lite
go get: added go.mongodb.org/mongo-driver v1.1.2
go get: added go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go get: added golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go get: added golang.org/x/net v0.0.0-20200625001655-4c5254603344
go get: added golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go get: added golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go get: added golang.org/x/text v0.3.2
go get: added golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go get: added google.golang.org/appengine v1.5.0
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.3.0
go get: added gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go get: added k8s.io/api v0.18.10
go get: added k8s.io/apimachinery v0.18.10
go get: added k8s.io/client-go v0.18.10
go get: added k8s.io/klog v1.0.0
go get: added k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go get: added k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go get: added sigs.k8s.io/kustomize/api v0.6.5
go get: added sigs.k8s.io/kustomize/cmd/config v0.8.5
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go get: added sigs.k8s.io/kustomize/kyaml v0.9.4
go get: added sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go get: added sigs.k8s.io/yaml v1.2.0
/root/gopath/src/kubebuilder-demo/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/redis.myapp.demo.kubebuilder.io created
[root@master kubebuilder-demo]

查看所创建的CRD

[root@master kubebuilder-demo]# kubectl get crds |grep redis
redis.myapp.demo.kubebuilder.io                       2022-02-13T11:32:20Z
[root@master kubebuilder-demo]# 

4、业务开发

本次实践中,我们将创建自定义资源Redis,实现的功能

(1)创建CRD

(2)创建Kind :Redis

(3)创建POD,支持副本数控制,验证

(3)创建多副本POD,支持副本收缩

为了开发方便,可以将上面的生成的“kubebuilder-demo”文件拷贝到windows下,并导入到goland中。并配置SFTP自动上传到$GOPATH/src/kubebuilder-demo下

另外也可以直接将文件拷贝到本地后,直接在本地运行

  • (1)将Kubernetes的~/.kube/config文件,拷贝到windows的$ %HOME% /.kube/下
    如:C:\Users\Administrator\.kube,之所以要这个做是因为kubebuilder中加载kubeconfig文件的顺序所决定的。详情见:Kubebuilder认证配置文件的加载
  • (2)针对于_types.go文件的修改,需要重新编译生成“zz_generated.deepcopy.go”文件,可以在Linux上make install后再拷贝回来

验证

Kubebuilder中的验证是通过标签的方式来完成,标签的基本形式为://+some-tag 或//+some-other-tag=value

标签分为两种:

  • 声明在package行上的全局标签
  • 在类型声明前的局部标签(如在一个struct定义前)

(1)redis_types.go,修改Foo为Port

	// +kubebuilder:validation:Maximum:=6380
	// +kubebuilder:validation:Minimum:=6370

	Port int `json:"port,omitempty"`

这里我们只是针对于port的范围做验证,有关Kubernetes的验证见:https://book.kubebuilder.io/reference/markers/crd-validation.html

(2)修改“myapp_v1_redis.yaml”文件,增加端口字段:

apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
  name: redis-sample
spec:
  # TODO(user): Add fields here
  port: 6379

由于这里我们修改了Redis struct的字段定义,所以需要重新make install

此时,我们可以看到自己的CRD中关于port字段的验证规则:

[root@master ~]# kubectl get crds |grep redis
redis.myapp.demo.kubebuilder.io                       2022-02-13T11:32:20Z
[root@master ~]# kubectl get crds/redis.myapp.demo.kubebuilder.io  -oyaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.8.0
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.8.0"},"creationTimestamp":null,"name":"redis.myapp.demo.kubebuilder.io"},"spec":{"group":"myapp.demo.kubebuilder.io","names":{"kind":"Redis","listKind":"RedisList","plural":"redis","singular":"redis"},"scope":"Namespaced","versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"Redis is the Schema for the redis API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"RedisSpec defines the desired state of Redis","properties":{"num":{"type":"integer"},"port":{"maximum":6380,"minimum":6370,"type":"integer"}},"type":"object"},"status":{"description":"RedisStatus defines the observed state of Redis","type":"object"}},"type":"object"}},"served":true,"storage":true,"subresources":{"status":{}}}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
  creationTimestamp: "2022-02-13T11:32:20Z"
  generation: 4
  name: redis.myapp.demo.kubebuilder.io
  resourceVersion: "510294"
  uid: 91122c22-e52e-46cb-bcde-0a5b20578bc2
spec:
  conversion:
    strategy: None
  group: myapp.demo.kubebuilder.io
  names:
    kind: Redis
    listKind: RedisList
    plural: redis
    singular: redis
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Redis is the Schema for the redis API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: RedisSpec defines the desired state of Redis
            properties:
              num:
                type: integer
              port:
                maximum: 6380 #验证规则
                minimum: 6370
                type: integer
            type: object
          status:
            description: RedisStatus defines the observed state of Redis
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: Redis
    listKind: RedisList
    plural: redis
    singular: redis
  conditions:
  - lastTransitionTime: "2022-02-13T11:32:20Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2022-02-13T11:32:20Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1
[root@master ~]

(3)应用“myapp_v1_redis.yaml”文件,创建Kind Redis

[root@master kubebuilder-demo]# kubectl apply -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample configured

[root@master ~]# kubectl get Redis
NAME           AGE
redis-sample   95m

上面就显示了NAME和AGE,实际上这里的显示格式,也是可以自定义的。

(4)修改“myapp_v1_redis.yaml”的port为6382,能够看到报错

[root@master kubebuilder-demo]# kubectl apply -f  config/samples/myapp_v1_redis.yaml 
The Redis "redis-sample" is invalid: spec.port: Invalid value: 6382: spec.port in body should be less than or equal to 6380

创建POD

创建“controllers/redis_helper.go”文件:

package controllers

import (
	"context"
	coreV1 "k8s.io/api/core/v1"
	v1 "kubebuilder-demo/api/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

func CreateRedis(client client.Client, redisConfig *v1.Redis) error {
	newPod := &coreV1.Pod{}
	newPod.Name = redisConfig.Name
	newPod.Namespace = redisConfig.Namespace
	newPod.Spec.Containers = []coreV1.Container{
		{
			Name:            redisConfig.Name,
			Image:           "redis:5-alpine",
			ImagePullPolicy: coreV1.PullIfNotPresent,
			Ports: []coreV1.ContainerPort{
				{
					ContainerPort: int32(redisConfig.Spec.Port),
				},
			},
		},
	}
	return client.Create(context.Background(), newPod)
}

修改“controllers/redis_controller.go”文件的Reconcile,增加如下逻辑:

func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	redis := &myappv1.Redis{}
	err := r.Get(ctx, req.NamespacedName, redis)
	if err != nil {
		fmt.Println("error:", err)
	} else {
		fmt.Println("redis Obj:", redis)
		err := CreateRedis(r.Client, redis)
		fmt.Println("create pod failue,", err)
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
}

此时make run,并打开另外一个窗口执行“kubectl apply -f config/samples/myapp_v1_redis.yaml”

注意:为了不影响结果,需要删除前面一个步骤生成的Kind;kubectl delete -f

[root@master kubebuilder-demo]# kubectl get Redis 
No resources found in default namespace.
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]# kubectl apply -f config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get Redis 
NAME           AGE
redis-sample   21s
[root@master kubebuilder-demo]# kubectl get pods #已经创建了pod
NAME           READY   STATUS    RESTARTS   AGE
redis-sample   1/1     Running   0          16s
[root@master kubebuilder-demo]# 

注意:这里存在一个问题,如果我们将创建资源的CRD文件删掉,则POD依旧存在:

[root@master kubebuilder-demo]# kubectl delete -f config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io "redis-sample" deleted
[root@master kubebuilder-demo]# kubectl get pods #pod依旧存在
NAME           READY   STATUS    RESTARTS   AGE
redis-sample   1/1     Running   0          72s
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]# 

这就造成了所创建POD的孤立存在。

为此,我们可以使用finalizers来限制,如果POD没有删除,不能删除对应POD。

有关finalizers的使用见:https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/

另外需要说明的是,POD究竟是如何被创建出来的,追踪代码能够发现实际上调用的就是Kubernetes的API:

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#create-pod-v1-core

HTTP Request

POST /api/v1/namespaces/{namespace}/pods
Parameter Description
namespace object name and auth scope, such as for teams and projects

Query Parameters

Parameter Description
pretty If 'true', then the output is pretty printed.
dryRun When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
fieldManager fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.

Body Parameters

Parameter Description
body Pod

Response

Code Description
200 Pod OK
201 Pod Created
202 Pod Accepte

跟踪代码“client.Create(context.Background(), newPod)”,

sigs.k8s.io/controller-runtime@v0.11.0/pkg/client/client.go

// Create implements client.Client.
func (c *client) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
	switch obj.(type) {
	case *unstructured.Unstructured:
		return c.unstructuredClient.Create(ctx, obj, opts...)
	case *metav1.PartialObjectMetadata:
		return fmt.Errorf("cannot create using only metadata")
	default:
		return c.typedClient.Create(ctx, obj, opts...)//
	}
}

// Create implements client.Client.
func (c *typedClient) Create(ctx context.Context, obj Object, opts ...CreateOption) error {
	o, err := c.cache.getObjMeta(obj)
	if err != nil {
		return err
	}

	createOpts := &CreateOptions{}
	createOpts.ApplyOptions(opts)
	return o.Post().
		NamespaceIfScoped(o.GetNamespace(), o.isNamespaced()).
		Resource(o.resource()).
		Body(obj).
		VersionedParams(createOpts.AsCreateOptions(), c.paramCodec).
		Do(ctx).
		Into(obj)
}

// request connects to the server and invokes the provided function when a server response is
// received. It handles retry behavior and up front validation of requests. It will invoke
// fn at most once. It will return an error if a problem occurred prior to connecting to the
// server - the provided function is responsible for handling server errors.
func (r *Request) request(ctx context.Context, fn func(*http.Request, *http.Response)) error {
	//Metrics for total request latency
	start := time.Now()
	defer func() {
		metrics.RequestLatency.Observe(ctx, r.verb, r.finalURLTemplate(), time.Since(start))
	}()

	if r.err != nil {
		klog.V(4).Infof("Error in request: %v", r.err)
		return r.err
	}

	if err := r.requestPreflightCheck(); err != nil {
		return err
	}

	client := r.c.Client
	if client == nil {
		client = http.DefaultClient
	}

	// Throttle the first try before setting up the timeout configured on the
	// client. We don't want a throttled client to return timeouts to callers
	// before it makes a single request.
	if err := r.tryThrottle(ctx); err != nil {
		return err
	}

	if r.timeout > 0 {
		var cancel context.CancelFunc
		ctx, cancel = context.WithTimeout(ctx, r.timeout)
		defer cancel()
	}

	// Right now we make about ten retry attempts if we get a Retry-After response.
	var retryAfter *RetryAfter
	for {
		req, err := r.newHTTPRequest(ctx)
		if err != nil {
			return err
		}

		r.backoff.Sleep(r.backoff.CalculateBackoff(r.URL()))
		if retryAfter != nil {
			// We are retrying the request that we already send to apiserver
			// at least once before.
			// This request should also be throttled with the client-internal rate limiter.
			if err := r.tryThrottleWithInfo(ctx, retryAfter.Reason); err != nil {
				return err
			}
			retryAfter = nil
		}
		resp, err := client.Do(req) //在这里调用http.client发送请求报文
		updateURLMetrics(ctx, r, resp, err)
		....
    } 
}

最终在这里完成发送请求,将POD放入到请求荷载中:

net/http/client.go:725

创建多副本的POD

(1)修改“api/v1/redis_types.go”文件,增加副本Num字段:

// RedisSpec defines the desired state of Redis
type RedisSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of Redis. Edit redis_types.go to remove/update

	// +kubebuilder:validation:Maximum:=6380
	// +kubebuilder:validation:Minimum:=6370

	Port int `json:"port,omitempty"`

	Num int `json:"num,omitempty"`
}

(2)修改“controllers/redis_helper.go”:

package controllers

import (
	"context"
	"fmt"
	coreV1 "k8s.io/api/core/v1"
	v1 "kubebuilder-demo/api/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
)
//判断对应名称的pod在finalizer中是否存在
func isExist(podName string, redis *v1.Redis) bool {
	for _, finalizer := range redis.Finalizers {
		if podName == finalizer {
			return true
		}
	}
	return false
}
//实际创建POD
func CreateRedis(podName string, client client.Client, redisConfig *v1.Redis) (string, error) {
	if isExist(podName, redisConfig) {
		return "", nil
	}
	newPod := &coreV1.Pod{}
	newPod.Name = podName
	newPod.Namespace = redisConfig.Namespace
	newPod.Spec.Containers = []coreV1.Container{
		{
			Name:            redisConfig.Name,
			Image:           "redis:5-alpine",
			ImagePullPolicy: coreV1.PullIfNotPresent,
			Ports: []coreV1.ContainerPort{
				{
					ContainerPort: int32(redisConfig.Spec.Port),
				},
			},
		},
	}
	return podName, client.Create(context.Background(), newPod)
}
//为多副本POD命名
func GetRedisPodNames(redis *v1.Redis) []string {
	podNames := make([]string, redis.Spec.Num)
	for i := 0; i < redis.Spec.Num; i++ {
		podNames[i] = fmt.Sprintf("%s-%d", redis.Name, i)
	}

	return podNames
}

(3)修改“controllers/redis_controller.go”:

// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.0/pkg/reconcile
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	redis := &myappv1.Redis{}
	err := r.Get(ctx, req.NamespacedName, redis)
	if err != nil {
		return ctrl.Result{}, err
	}

	fmt.Println("redis Obj:", redis)
    //取得需要创建的POD副本的名称
	podNames := GetRedisPodNames(redis)
	fmt.Println("podNames,", podNames)
	updateFlag := false

    //删除POD,删除Kind:Redis过程中,会自动加上DeletionTimestamp字段,据此判断是否删除了自定义资源
	if !redis.DeletionTimestamp.IsZero() {
		return ctrl.Result{}, r.clearRedis(ctx, redis)
	}

	//创建POD
	for _, podName := range podNames {
		finalizerPodName, err := CreateRedis(podName, r.Client, redis)
		if err != nil {
			fmt.Println("create pod failue,", err)
			return ctrl.Result{}, err
		}
		if finalizerPodName == "" {
			continue
		}
        //若该pod已经不在finalizers中,则添加
		redis.Finalizers = append(redis.Finalizers, finalizerPodName)
		updateFlag = true
	}
	//更新Kind Redis状态
	if updateFlag {
		err = r.Client.Update(ctx, redis)
		if err != nil {
			return ctrl.Result{}, err
		}
	}
	return ctrl.Result{}, nil
}
//删除POD逻辑
func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {
    //从finalizers中取出podName,然后执行删除
	for _, finalizer := range redis.Finalizers {
        //删除pod
		err := r.Client.Delete(ctx, &v1.Pod{
			ObjectMeta: metav1.ObjectMeta{
				Name:      finalizer,
				Namespace: redis.Namespace,
			},
		})
		if err != nil {
			return err
		}
	}
    //清空finializers,只要它有值,就无法删除Kind
	redis.Finalizers = []string{}
	return r.Client.Update(ctx, redis)
}

(4)修改“config/samples/myapp_v1_redis.yaml”,增加“num”

apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
  name: redis-sample
spec:
  # TODO(user): Add fields here
  port: 6379
  num: 2

(5)make install后make run,在另外一个窗口中执行kubectl apply -f

注意:每次修改type类型定义文件,都需要进行make install

[root@master kubebuilder-demo]# kubectl apply -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get Redis
NAME           AGE
redis-sample   6s
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          10s
redis-sample-1   1/1     Running   0          10s
[root@master kubebuilder-demo]#

此时查看Kind:Redis的yaml文件,能够看到finalizers字段上填充了两个POD的名字

[root@master kubebuilder-demo]# kubectl get Redis/redis-sample -oyaml 
apiVersion: myapp.demo.kubebuilder.io/v1
kind: Redis
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"myapp.demo.kubebuilder.io/v1","kind":"Redis","metadata":{"annotations":{},"name":"redis-sample","namespace":"default"},"spec":{"num":2,"port":6379}}
  creationTimestamp: "2022-02-14T15:13:09Z"
  finalizers:  #字段填充了POD名称
  - redis-sample-0
  - redis-sample-1
  generation: 1
  name: redis-sample
  namespace: default
  resourceVersion: "561478"
  uid: 46a36e7c-2af9-47b8-a882-1a9c11ab7110
spec:
  num: 2
  port: 6379

(6)此时尝试删除Kind:Redis,对应的POD也会被删除

[root@master kubebuilder-demo]# kubectl delete -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io "redis-sample" deleted
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS        RESTARTS   AGE
redis-sample-0   0/1     Terminating   0          88s
redis-sample-1   0/1     Terminating   0          88s
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]

(7)修改“config/samples/myapp_v1_redis.yaml”的num为1,并再次应用

查看POD,能够发现POD数,并未发生变化:

[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          4m27s
redis-sample-1   1/1     Running   0          4m27s
[root@master kubebuilder-demo]# 

副本收缩

(1)修改“controllers/redis_controller.go”,优化副本删除逻辑:

func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	redis := &myappv1.Redis{}
	err := r.Get(ctx, req.NamespacedName, redis)
	if err != nil {
		return ctrl.Result{}, err
	}

	fmt.Println("redis Obj:", redis)

	//(1)删除Kind Redis时,删除所创建的POD
	//(2)副本收缩时,缩减POD
	if !redis.DeletionTimestamp.IsZero() || len(redis.Finalizers) > redis.Spec.Num {
		return ctrl.Result{}, r.clearRedis(ctx, redis)
	}

	//创建POD
	podNames := GetRedisPodNames(redis)
	fmt.Println("podNames,", podNames)
	updateFlag := false
	for _, podName := range podNames {
		finalizerPodName, err := CreateRedis(podName, r.Client, redis)
		if err != nil {
			fmt.Println("create pod failue,", err)
			return ctrl.Result{}, err
		}
		if finalizerPodName == "" {
			continue
		}
		redis.Finalizers = append(redis.Finalizers, finalizerPodName)
		updateFlag = true
	}
	//更新Kind Redis状态
	if updateFlag {
		err = r.Client.Update(ctx, redis)
		if err != nil {
			return ctrl.Result{}, err
		}
	}
	return ctrl.Result{}, nil
}

func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {

	//副本数和当前的num数量的差值,要删除的就是这个差值部分,
	//如果两则相等,则删除全部
	//情形如下:
	//(1)finalizers > num  可能出现,删除差值部分
	//(2)finalizers = num 可能出现,删除全部
	//(3)finalizers < num 不可能出现
	var deletedPodNames []string

	//后项删除,即:删除finalizers切片的最后的指定元素
	position := redis.Spec.Num
	if (len(redis.Finalizers) - redis.Spec.Num) != 0 {
		deletedPodNames = redis.Finalizers[position:]
		redis.Finalizers = redis.Finalizers[:position]
	} else {
		deletedPodNames = redis.Finalizers[:]
		redis.Finalizers = []string{}
	}

	fmt.Println("deletedPodNames", deletedPodNames)
	fmt.Println("redis.Finalizers", redis.Finalizers)
	for _, finalizer := range deletedPodNames {
		err := r.Client.Delete(ctx, &v1.Pod{
			ObjectMeta: metav1.ObjectMeta{
				Name:      finalizer,
				Namespace: redis.Namespace,
			},
		})
		if err != nil {
			return err
		}
	}
	return r.Client.Update(ctx, redis)
}

(2)执行make run,修改“config/samples/myapp_v1_redis.yaml”的num值为5,创建5个副本的POD

[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          4m27s
redis-sample-1   1/1     Running   0          4m27s
[root@master kubebuilder-demo]# vi config/samples/myapp_v1_redis.yaml 
[root@master kubebuilder-demo]# kubectl apply -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample configured
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS              RESTARTS   AGE
redis-sample-0   1/1     Running             0          12m
redis-sample-1   1/1     Running             0          12m
redis-sample-2   0/1     ContainerCreating   0          2s
redis-sample-3   0/1     ContainerCreating   0          2s
redis-sample-4   0/1     ContainerCreating   0          2s
[root@master kubebuilder-demo]#

等待POD创建就绪后,再次修改num为2,查看:

[root@master kubebuilder-demo]# kubectl apply -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample configured
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS        RESTARTS   AGE
redis-sample-0   1/1     Running       0          14m
redis-sample-1   1/1     Running       0          14m
redis-sample-2   0/1     Terminating   0          2m5s
redis-sample-3   0/1     Terminating   0          2m5s
redis-sample-4   0/1     Terminating   0          2m5s
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          15m
redis-sample-1   1/1     Running   0          15m
[root@master kubebuilder-demo]# 

删除kind:Redis自定义资源:

[root@master kubebuilder-demo]# kubectl delete -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io "redis-sample" deleted
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS        RESTARTS   AGE
redis-sample-0   0/1     Terminating   0          16m
redis-sample-1   0/1     Terminating   0          16m
[root@master kubebuilder-demo]# kubectl get pods
No resources found in default namespace.
[root@master kubebuilder-demo]# kubectl get Redis
No resources found in default namespace.
[root@master kubebuilder-demo]# 

POD删除和重建

在前面的POD创建时,我们没有使用Deployment来管理POD,一旦有些POD被删除掉,则不会自动创建。这里我们借助于Watches来实现,同时引入Owner Referenc概念,从中取出Kind和Name,据此来判断删除的是我们POD,然后在等待队列中,增加“reconcile.Request”,以此来实现对于副本数量的控制。

关于Owner Reference:

https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/

Owner References

Owner references describe how groups of objects are related. They are properties on resources that specify the relationship to one another, so entire trees of resources can be deleted.

Finalizer rules are processed when there are owner references. An owner reference consists of a name and a UID. Owner references link resources within the same namespace, and it also needs a UID for that reference to work. Pods typically have owner references to the owning replica set. So, when deployments or stateful sets are deleted, then the child replica sets and pods are deleted in the process.

(1)修改“controllers/redis_controller.go”,监听POD:

//POD删除时的回调
func (r *RedisReconciler) poddeleteHandler(event event.DeleteEvent, limitingInterface workqueue.RateLimitingInterface) {
	fmt.Println("deleted pod name is :", event.Object.GetName())
    //取得OwnerReference,如果Kind和Name匹配,则触发reconcile.Request,并加入到等待队列
	for _, ownerReference := range event.Object.GetOwnerReferences() {
		if ownerReference.Kind == "Redis" && ownerReference.Name == "redis-sample" {
			limitingInterface.Add(reconcile.Request{
				NamespacedName: types.NamespacedName{
					Namespace: event.Object.GetNamespace(),
					Name:      ownerReference.Name,
				},
			})
		}
	}
}

// SetupWithManager sets up the controller with the Manager.
func (r *RedisReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&myappv1.Redis{}).
		Watches(&source.Kind{ 
			Type: &v1.Pod{},
		}, handler.Funcs{
			DeleteFunc: r.poddeleteHandler,
		}).
		Complete(r)

}

(2)修改“controllers/redis_helper.go”

//优化判断POD存在逻辑
func isExistPod(podName string, redis *v1.Redis, client client.Client) bool {
	err := client.Get(context.Background(), types.NamespacedName{
		Name:      podName,
		Namespace: redis.Namespace,
	}, &coreV1.Pod{})

	if err != nil {
		return false
	}
	return true
}

func CreateRedis(podName string, client client.Client, redisConfig *v1.Redis, scheme *runtime.Scheme) (string, error) {
	if isExistPod(podName, redisConfig, client) {
		return podName, nil
	}
	newPod := &coreV1.Pod{}
	newPod.Name = podName
	newPod.Namespace = redisConfig.Namespace
	newPod.Spec.Containers = []coreV1.Container{
		{
			Name:            redisConfig.Name,
			Image:           "redis:5-alpine",
			ImagePullPolicy: coreV1.PullIfNotPresent,
			Ports: []coreV1.ContainerPort{
				{
					ContainerPort: int32(redisConfig.Spec.Port),
				},
			},
		},
	}
	err := controllerutil.SetControllerReference(redisConfig, newPod, scheme) //为POD增加OwnerReference
	if err != nil {
		return podName, err
	}
	return podName, client.Create(context.Background(), newPod)
}

(3)Make run

#删除旧的自定义资源
[root@master kubebuilder-demo]# kubectl delete -f  config/samples/myapp_v1_redis.yaml

[root@master kubebuilder-demo]# kubectl apply  -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          13s
redis-sample-1   1/1     Running   0          13s
[root@master kubebuilder-demo]# kubectl delete pod redis-sample-1
pod "redis-sample-1" deleted
#在另外一个窗口查看pod情况
[root@master ~]# kubectl get pods
NAME             READY   STATUS        RESTARTS   AGE
redis-sample-0   1/1     Running       0          19s
redis-sample-1   0/1     Terminating   0          90s
[root@master ~]
#pod又被重新创建
[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          37s
redis-sample-1   1/1     Running   0          8s
[root@master kubebuilder-demo]#

查看POD的yaml,观察OwnerReference字段:

[root@master kubebuilder-demo]# kubectl get pod redis-sample-0 -oyaml 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 00e0b8f104de05271f3b0b9aa2f72e8bf6e919ce5139da10ad1c76e215fd4365
    cni.projectcalico.org/podIP: 10.233.70.64/32
    cni.projectcalico.org/podIPs: 10.233.70.64/32
  creationTimestamp: "2022-02-15T09:11:21Z"
  name: redis-sample-0
  namespace: default
  ownerReferences:
  - apiVersion: myapp.demo.kubebuilder.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Redis
    name: redis-sample
    uid: ed99405e-df1e-4d24-90f8-5994aa4c7f41
  resourceVersion: "639304"
  uid: 3173af40-498e-4d79-a006-cebaccdf86a9
spec:
  ...

其中“ownerReferences”中“ uid: ed99405e-df1e-4d24-90f8-5994aa4c7f41”为自定义资源Redis的UID:

[root@master kubebuilder-demo]# kubectl get Redis -oyaml
apiVersion: v1
items:
- apiVersion: myapp.demo.kubebuilder.io/v1
  kind: Redis
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"myapp.demo.kubebuilder.io/v1","kind":"Redis","metadata":{"annotations":{},"name":"redis-sample","namespace":"default"},"spec":{"num":2,"port":6379}}
    creationTimestamp: "2022-02-15T09:09:38Z"
    finalizers:
    - redis-sample-0
    - redis-sample-1
    generation: 1
    name: redis-sample
    namespace: default
    resourceVersion: "639070"
    uid: ed99405e-df1e-4d24-90f8-5994aa4c7f41
  spec:
    num: 2
    port: 6379
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

添加事件支持

目前在自定义资源Kind:Redis中Events:

[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
Name:         redis-sample
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  myapp.demo.kubebuilder.io/v1
Kind:         Redis
Metadata:
  Creation Timestamp:  2022-02-15T09:09:38Z
  Finalizers:
    redis-sample-0
    redis-sample-1
  Generation:  1
  Managed Fields:
    API Version:  myapp.demo.kubebuilder.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:num:
        f:port:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-02-15T09:09:38Z
    API Version:  myapp.demo.kubebuilder.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"redis-sample-0":
          v:"redis-sample-1":
    Manager:         main
    Operation:       Update
    Time:            2022-02-15T09:09:38Z
  Resource Version:  639070
  UID:               ed99405e-df1e-4d24-90f8-5994aa4c7f41
Spec:
  Num:   2
  Port:  6379
Events:  <none>
[root@master kubebuilder-demo]# 

现在需要增加对于增加和删除POD的事件进行记录。

(1)修改redis_controller.go

// RedisReconciler reconciles a Redis object
type RedisReconciler struct {
	client.Client
	Scheme *runtime.Scheme
	EventRecorder record.EventRecorder  //增加事件记录
}

func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	...
	//更新Kind Redis状态
	if updateFlag {
		r.EventRecorder.Event(redis,v1.EventTypeNormal,"Upgrade","scale replicates")//创建POD时,记录事件
		err = r.Client.Update(ctx, redis)
		if err != nil {
			return ctrl.Result{}, err
		}
	}
	return ctrl.Result{}, nil
}


func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {

	...
    r.EventRecorder.Event(redis,v1.EventTypeNormal,"Updated","scale replicates")//删除POD时,记录事件
	return r.Client.Update(ctx, redis)
}

(2)修改main.go,创建RedisReconciler时,为EventRecorder属性赋值

func main() {
	...
	if err = (&controllers.RedisReconciler{
		Client:        mgr.GetClient(),
		Scheme:        mgr.GetScheme(),
		EventRecorder: mgr.GetEventRecorderFor("demo.kubebuilder.io"), //名称任意
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "Redis")
		os.Exit(1)
	}
	...
}

(3)make install & make run ,然后测试效果

[root@master kubebuilder-demo]# kubectl apply  -f  config/samples/myapp_v1_redis.yaml 
redis.myapp.demo.kubebuilder.io/redis-sample created
[root@master kubebuilder-demo]# kubectl get Redis
NAME           AGE
redis-sample   29s

[root@master kubebuilder-demo]# kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
redis-sample-0   1/1     Running   0          80s
redis-sample-1   1/1     Running   0          80s
[root@master kubebuilder-demo]# 

[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
Name:         redis-sample
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  myapp.demo.kubebuilder.io/v1
Kind:         Redis
Metadata:
  Creation Timestamp:  2022-02-15T10:14:38Z
  Finalizers:
    redis-sample-0
    redis-sample-1
  Generation:  1
  Managed Fields:
    API Version:  myapp.demo.kubebuilder.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:num:
        f:port:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-02-15T10:14:38Z
    API Version:  myapp.demo.kubebuilder.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"redis-sample-0":
          v:"redis-sample-1":
    Manager:         main
    Operation:       Update
    Time:            2022-02-15T10:14:38Z
  Resource Version:  643906
  UID:               aae17d51-0889-40d0-838d-d6f18d411956
Spec:
  Num:   2
  Port:  6379
Events:
  Type    Reason   Age   From                 Message
  ----    ------   ----  ----                 -------
  Normal  Upgrade  40s   demo.kubebuilder.io  scale replicates 

修改副本数为1:

[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
...
Events:
  Type    Reason   Age                    From                 Message
  ----    ------   ----                   ----                 -------
  Normal  Upgrade  4m59s (x2 over 7m39s)  demo.kubebuilder.io  scale replicates
  Normal  Updated  18s (x2 over 65s)      demo.kubebuilder.io  scale replicates

额外信息的输出

默认情况下,我们查看自定义资源Redis时,输出的是这样的:

[root@master kubebuilder-demo]# kubectl get Redis 
NAME           AGE
redis-sample   21s

如果我们想要输出中包含副本的数量,即:

[root@master kubebuilder-demo]# kubectl get Redis
NAME           REPLICAS
redis-sample   5

关于Addition Printer Columns

Additional Printer Columns

Starting with Kubernetes 1.11, kubectl get can ask the server what columns to display. For CRDs, this can be used to provide useful, type-specific information with kubectl get, similar to the information provided for built-in types.

The information that gets displayed can be controlled with the additionalPrinterColumns field on your CRD, which is controlled by the +kubebuilder:printcolumn marker on the Go type for your CRD.

For instance, in the following example, we add fields to display information about the knights, rank, and alias fields from the validation example:

// +kubebuilder:printcolumn:name="Alias",type=string,JSONPath=`.spec.alias`
// +kubebuilder:printcolumn:name="Rank",type=integer,JSONPath=`.spec.rank`
// +kubebuilder:printcolumn:name="Bravely Run Away",type=boolean,JSONPath=`.spec.knights[?(@ == "Sir Robin")]`,description="when danger rears its ugly head, he bravely turned his tail and fled",priority=10
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
type Toy struct {
   metav1.TypeMeta   `json:",inline"`
   metav1.ObjectMeta `json:"metadata,omitempty"`

   Spec   ToySpec   `json:"spec,omitempty"`
   Status ToyStatus `json:"status,omitempty"`
}

这里需要使用到“+kubebuilder:printcolumn”,关于它的字段描述:

https://book.kubebuilder.io/reference/markers/crd.html

(1)修改“api/v1/redis_types.go”

type RedisStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	Replicas int `json:"replicas,omitempty"` //添加副本显示
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:printcolumn:JSONPath=".status.replicas",name=Replicas,type=integer

// Redis is the Schema for the redis API
type Redis struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   RedisSpec   `json:"spec,omitempty"`
	Status RedisStatus `json:"status,omitempty"`
}

(2)修改“controllers/redis_controller.go”

func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	...
	//更新Kind Redis状态
	if updateFlag {
         ...
		//更新status状态值
		redis.Status.Replicas = len(redis.Finalizers)
		err := r.Status().Update(ctx, redis)
		if err != nil {
			return ctrl.Result{}, err
		}
	}
	return ctrl.Result{}, nil
}


func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {

	...
	//更新status状态值
	redis.Status.Replicas = len(redis.Finalizers)
	err = r.Status().Update(ctx, redis)
	if err != nil {
		return err
	}
	return nil
}

(3)验证

make install 后make run

[root@master kubebuilder-demo]# kubectl get Redis
NAME           REPLICAS
redis-sample   5
[root@master kubebuilder-demo]# kubectl describe Redis/redis-sample
Name:         redis-sample
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  myapp.demo.kubebuilder.io/v1
Kind:         Redis
...
Spec:
  Num:   5
  Port:  6379
Status:
  Replicas:  5
Events:
  Type    Reason   Age   From                 Message
  ----    ------   ----  ----                 -------
  Normal  Upgrade  28s   demo.kubebuilder.io  scale replicates

本实例代码地址:https://gitee.com/cosmoswong/kubebuilder-demo.git

posted @ 2022-02-14 23:32  cosmoswong  阅读(2586)  评论(0编辑  收藏  举报