Operator介绍
一、Operator简介
在Kubernetes中我们经常使用Deployment、DaemonSet、Service、ConfigMap等资源，这些资源都是Kubernetes的内置资源，他们的创建、更新、删除等均有Controller Manager负责管理。
二、Operator组成
Operator(Controller+CRD），Operator是由Kubernetes自定义资源（CRD）和控制器（Controller）构成的元原生扩展服务，其中CRD定义了每个Operator需要创建和管理的自定义资源对象，底层实际就是通过APIServer接口在ETCD中注册一种新的资源类型，注册完成后就可以创建该资源类型的对象了。但是仅注册资源和创建资源对象是没有任何实际意义的，CRD最重要的是需要配合对应的Controller来实现自定义资源的功能达到自定义资源期望的状态，比如内置的Deployment Controller用来控制Department资源对象的功能，根据配置生成特定数量的Pod监控其状态，并根据事件做出相应的动作。
三、Operator使用
用户想为自己的自定义资源构建一个Kubernetes Operator，有很多工具可供选择比如Operator SDK、Kubebuilder，甚至可以使用Operator SDK（HELM、Ansible、Go）。这些工具创建Kubernetes Operator用来监控自定义资源，并且根据资源的变化调整资源状态。
Operator作为自定义扩展资源以Deployment的方式部署到Kubernetes中，通过List-Watch方式监听对应资源的变化，当用户修改自定义资源中的任何内容，Operator会监控资源的更改，并根据更改内容执行特定的操作，这些操作通常会对Kubernetes API中某些资源进行调用。
四、Operator应用案例
Kubebuilder介绍
Kubebuilder是一个用Go语言构建Kubenetes API控制器和CRD的脚手架工具，通过使用Kubebuilder,用户可以遵循一套简单的编程框架，编写Operator使用实例。
- kubenetes: v1.23.8
- go
tar xf go1.18.3.linux-amd64.tar.gz
mv go /usr/local/
vim /etc/profile.d/go183.sh
export GO111MODULE=on
export GOROOT=/usr/local/go
export GOPATH=/root/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
source /etc/profile.d/go183.sh- kubebuilder: 3.5.0
yum -y install gcc
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/
kubebuilder version案例
Welcome案例主要实现使用Operator和CRD部署一套完整的应用环境，可以实现根据自定义类型创建资源，通过创建一个Welcome类型的资源，后台自动创建Deployment和Service，通过Web页面访问Service呈现应用部署，通过自定义控制器方式进行控制管理
我们西药创建Welcome自定义资源及对应的Controllers，最终我们可以通过类似如下代码的YAML文件部署简单的Web应用
apiVersion: webapp.demo.welcome.domain/v1
kind: Welcome
metadata:
name: welcome-sample
spec:
name: myfriendsWeb应用介绍
本案例中，我们使用Go语言HTTP模块创建一个Web服务，用户访问页面后会自动加载NAME及PORT环境变量并渲染index.html静态文件中，代码如下：
package main
import (
"fmt"
"net/http"
"os"
)
func main() {
name := os.Getenv("NAME")
hello := fmt.Sprintf("Hello %s", name)
http.Handle("/hello/", http.StripPrefix("/hello/", http.FileServer(http.Dir("static"))))
f, err := os.OpenFile("./static/index.html", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
panic(err)
}
defer f.Close()
if _, err = f.WriteString(hello); err != nil {
panic(err)
}
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
}其中，NAME环境变量通过我们在Welcome中定义的name字段获取，我们在下面的控制器编写中会详细介绍获取字段的详细方法。我们将index.html放在Static文件夹下，并将工程文件打包为Docker镜像，Dockerfile如下：
FROM golang:1.17 as builder
WORKDIR /
COPY . .
COPY static
RUN CGO_ENABLED=0 GOOS=linux go build -v -o main
FROM alpine
RUN apk add --no-cache ca-certificates
COPY --from=builder /main /usr/local/main
COPY --from=builder static /static
CMD ["/usr/local/main"]项目初始化
使用Kubebuilder命令镜像项目初始化
mkdir demo
cd demo
go mod init welcome_demo.domain
kubebuilder init --domain demo.welcome.domain初始化项目后，kubebuilder生成项目接口如下
[root@k8s-01 demo]# tree .
.
├── config
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ └── manager_config_patch.yaml
│ ├── manager
│ │ ├── controller_manager_config.yaml
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── prometheus
│ │ ├── kustomization.yaml
│ │ └── monitor.yaml
│ └── rbac
│ ├── auth_proxy_client_clusterrole.yaml
│ ├── auth_proxy_role_binding.yaml
│ ├── auth_proxy_role.yaml
│ ├── auth_proxy_service.yaml
│ ├── kustomization.yaml
│ ├── leader_election_role_binding.yaml
│ ├── leader_election_role.yaml
│ ├── role_binding.yaml
│ └── service_account.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md
6 directories, 25 files
创建`Welcome` Kind和其对应的控制器
[root@k8s-01 demo]# kubebuilder create api --group webapp --kind Welcome --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y输入两次y, Kubebuilder 分别创建了资源和控制器的模板，此处的group、version、kind这3个属性组合起来标识一个k8s的CRD，创建完成后，Kubebuilder添加文件如下：
[root@k8s-01 demo]# tree .
.
├── api
│ └── v1
│ ├── groupversion_info.go
│ ├── welcome_types.go
│ └── zz_generated.deepcopy.go
├── bin
│ └── controller-gen
├── config
│ ├── crd
│ │ ├── kustomization.yaml
│ │ ├── kustomizeconfig.yaml
│ │ └── patches
│ │ ├── cainjection_in_welcomes.yaml
│ │ └── webhook_in_welcomes.yaml
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ └── manager_config_patch.yaml
│ ├── manager
│ │ ├── controller_manager_config.yaml
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── prometheus
│ │ ├── kustomization.yaml
│ │ └── monitor.yaml
│ ├── rbac
│ │ ├── auth_proxy_client_clusterrole.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── role_binding.yaml
│ │ ├── service_account.yaml
│ │ ├── welcome_editor_role.yaml
│ │ └── welcome_viewer_role.yaml
│ └── samples
│ └── webapp_v1_welcome.yaml
├── controllers
│ ├── suite_test.go
│ └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md
13 directories, 38 files后续需要执行两步操作：
- 修改Resource Type
- 修改Controller 逻辑
修改Resource Type
此处Resource Type为需要定义的资源字段，用于在Yaml文件中进行声明，本案例中需要新增name字段用于“Welcome”Kind中的Web应用，代码如下：
vim api/v1/welcome_types.go
// WelcomeSpec defines the desired state of Welcome
type WelcomeSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
// Foo is an example field of Welcome. Edit welcome_types.go to remove/update
// Foo string `json:"foo,omitempty"`
Name string `json:"name,omitempty"`
}修改Controller逻辑
在Controller中需要通过Reconcile方法完成Deployment和Service部署，并最终达到期望的状态。
vim controllers/welcome_controller.go
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update
// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.1/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// _ = log.FromContext(ctx)
// TODO(user): your logic here
log := r.Log.WithValues("welcome", req.NamespacedName)
log.Info("reconcilling welcome")
return ctrl.Result{}, nil
}- 最终代码如下
/*
Copyright 2022.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package controllers
import (
"context"
"fmt"
appsv1 "k8s.io/api/apps/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/intstr"
"sigs.k8s.io/controller-runtime/pkg/log"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
webappv1 "welcome_demo.domain/api/v1"
)
// WelcomeReconciler reconciles a Welcome object
type WelcomeReconciler struct {
client.Client
Scheme *runtime.Scheme
}
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update
// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.1/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
// TODO(user): your logic here
welcome := &webappv1.Welcome{}
if err := r.Client.Get(ctx, req.NamespacedName, welcome); err != nil {
return ctrl.Result{}, err
}
deployment, err := r.createWelcomeDeployment(welcome)
if err != nil {
return ctrl.Result{}, err
}
fmt.Println("create deployment success!")
svc, err := r.createService(welcome)
if err != nil {
return ctrl.Result{}, err
}
fmt.Println("create service success!")
applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("welcome_controller")}
err = r.Patch(ctx, &deployment, client.Apply, applyOpts...)
if err != nil {
return ctrl.Result{}, err
}
err = r.Patch(ctx, &svc, client.Apply, applyOpts...)
if err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
func (r *WelcomeReconciler) createWelcomeDeployment(welcome *webappv1.Welcome) (appsv1.Deployment, error) {
defOne := int32(1)
name := welcome.Spec.Name
if name == "" {
name = "world"
}
depl := appsv1.Deployment{
TypeMeta: metav1.TypeMeta{
APIVersion: appsv1.SchemeGroupVersion.String(),
Kind: "Deployment",
},
ObjectMeta: metav1.ObjectMeta{
Name: welcome.Name,
Namespace: welcome.Namespace,
},
Spec: appsv1.DeploymentSpec{
Replicas: &defOne,
Selector: &metav1.LabelSelector{
MatchLabels: map[string]string{"welcome": welcome.Name},
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{"welcome": welcome.Name},
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "welcome",
Env: []corev1.EnvVar{
{Name: "NAME", Value: name},
},
Ports: []corev1.ContainerPort{
{ContainerPort: 8080,
Name: "http",
Protocol: "TCP",
},
},
Image: "sdfcdwefe/operatordemo:v1",
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: *resource.NewMilliQuantity(100, resource.DecimalSI),
corev1.ResourceMemory: *resource.NewMilliQuantity(100000, resource.BinarySI),
},
},
},
},
},
},
},
}
return depl, nil
}
func (r *WelcomeReconciler) createService(welcome *webappv1.Welcome) (corev1.Service, error) {
svc := corev1.Service{
TypeMeta: metav1.TypeMeta{
APIVersion: corev1.SchemeGroupVersion.String(),
Kind: "Service"},
ObjectMeta: metav1.ObjectMeta{
Name: welcome.Name,
Namespace: welcome.Namespace,
},
Spec: corev1.ServiceSpec{
Ports: []corev1.ServicePort{
{Name: "http",
Port: 8080,
Protocol: "TCP",
TargetPort: intstr.FromString("http")},
},
Selector: map[string]string{"welcome": welcome.Name},
Type: corev1.ServiceTypeLoadBalancer,
},
}
return svc, nil
}
// SetupWithManager sets up the controller with the Manager.
func (r *WelcomeReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&webappv1.Welcome{}).
Complete(r)
}
Welcome应部署
- 生成CRD资源
[root@k8s-01 demo]# make manifests
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
[root@k8s-01 demo]# tree .
.
├── api
│ └── v1
│ ├── groupversion_info.go
│ ├── welcome_types.go
│ └── zz_generated.deepcopy.go
├── bin
│ └── controller-gen
├── config
│ ├── crd
│ │ ├── bases
│ │ │ └── webapp.demo.welcome.domain_welcomes.yaml
│ │ ├── kustomization.yaml
│ │ ├── kustomizeconfig.yaml
│ │ └── patches
│ │ ├── cainjection_in_welcomes.yaml
│ │ └── webhook_in_welcomes.yaml
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ └── manager_config_patch.yaml
│ ├── manager
│ │ ├── controller_manager_config.yaml
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── prometheus
│ │ ├── kustomization.yaml
│ │ └── monitor.yaml
│ ├── rbac
│ │ ├── auth_proxy_client_clusterrole.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── role_binding.yaml
│ │ ├── role.yaml
│ │ ├── service_account.yaml
│ │ ├── welcome_editor_role.yaml
│ │ └── welcome_viewer_role.yaml
│ └── samples
│ └── webapp_v1_welcome.yaml
├── controllers
│ ├── suite_test.go
│ └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md
14 directories, 40 files
[root@k8s-01 demo]# cat config/crd/bases/webapp.demo.welcome.domain_welcomes.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.9.0
creationTimestamp: null
name: welcomes.webapp.demo.welcome.domain
spec:
group: webapp.demo.welcome.domain
names:
kind: Welcome
listKind: WelcomeList
plural: welcomes
singular: welcome
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: Welcome is the Schema for the welcomes API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: WelcomeSpec defines the desired state of Welcome
properties:
name:
description: Foo is an example field of Welcome. Edit welcome_types.go
to remove/update Foo string `json:"foo,omitempty"`
type: string
type: object
status:
description: WelcomeStatus defines the observed state of Welcome
type: object
type: object
served: true
storage: true
subresources:
status: {}- 创建Welcome类型资源
[root@k8s-01 demo]# kubectl create -f config/crd/bases/
customresourcedefinition.apiextensions.k8s.io/welcomes.webapp.demo.welcome.domain created
[root@k8s-01 demo]# kubectl create -f config/samples/webapp_v1_welcome.yaml
welcome.webapp.demo.welcome.domain/welcome-sample created- 使用`kubectl get crd`命令查看自定义对象
[root@k8s-01 demo]# kubectl get crd | grep welcome
welcomes.webapp.demo.welcome.domain 2022-06-25T09:10:37Z- 通过kubectl get welcome命令可以看到创建的welcome对象
[root@k8s-01 demo]# kubectl get welcome
NAME AGE
welcome-sample 2m20s此时CRD并不会完成任何工作，只是在ETCD中创建了一条记录，我们需要运行Controller才能帮助我们完成工作并最终达到welcome定义的状态。
[root@k8s-01 demo]# make run
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1/welcome_types.go
go vet ./...
go run ./main.go
1.6561485987622015e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.6561485987624757e+09 INFO setup starting manager
1.6561485987638762e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.656148598763948e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.656148598764167e+09 INFO Starting EventSource {"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "source": "kind source: *v1.Welcome"}
1.6561485987641926e+09 INFO Starting Controller {"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome"}
1.6561485988653958e+09 INFO Starting workers {"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "worker count": 1}
create deployment success!
create service success!以上方式在本地启动控制器，方便调试和验证
- 查看创建的deployment、service
[root@k8s-01 demo]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
welcome-sample 1/1 1 1 34s
[root@k8s-01 demo]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d2h
welcome-sample LoadBalancer 10.106.36.129 <pending> 8080:30037/TCP 39s
