Kubernetes——自定义资源类型(CRD)
自定义资源类型(CRD)
Kubernetes 系统的扩展和增强既包括扩展 API Server 所支持的资源类型及相关声明式功能的实现,以及消除集群的单点以实现集群的高可用等,也包括如何将系统增强为一个完整意义上的 PaaS 平台,并以 DevOps 文化为驱动改善工作流程等。
Kubernetes API 默认提供的众多的功能性资源类型可用于容器编排以解决多数场景中的编排需求。但是,有些场景也需要借助于额外的或更高级别的编排抽象,例如引入集群外部的一些服务并以资源对象的形式进行管理,或者把 Kubernetes 的多个标准资源对象合并为一个单一的更高级别的资源抽象等等。而这类的 API 扩展抽象通常也应该兼容 Kubernetes 系统的基本特性,如支持 kubectl 管理工具、CRUD 及 watch 机制、标签、etcd 存储、认证、授权、RBAC 及审计,等等,从而使得用户可将精力集中于构建业务逻辑本身。
目前,扩展 Kubernetes API 的常用方式有三种:
- 使用 CRD(CustomResourceDefinitions)自定义资源类型。
- 开发自定义的 API Server 并聚合至主 API Server。
- 定制扩展 Kubernetes 源码。
其中,CRD 最易用但限制较多,自定义 API Server 更富有弹性但代码工作量偏大,而仅在必须添加新的核心类型才能确保专用的 Kubernetes 集群功能正常时才应该定制系统源码。
CRD 并非设计用来取代 Kubernetes 的原生资源类型,而是用于补充一种简单易用的更为灵活和更高级别的自定义 API 资源的方式。虽然目前在功能上仍存在不少的局限,但对于大多数的需求场景来说,CRD 的表现已经足够好,因此在满足需求的前提下是首选的 API 资源类型扩展方案。
一、创建 CRD 对象
CRD 自 Kubernetes 1.7 版开始引入,并自 1.8 版起完全取代其前身 TPR(ThirdPartyResources),其设计目标是通过 CRD 可以向 API 中增加新资源类型,无须修改 Kubernetes 源代码就能扩展它支持使用 API 资源类型。该功能大大提高了 Kubernetes 的可扩展能力。
CRD 本身也是一种资源类型,是一种无须编码就可以扩展原生 Kubernetes API 接口的形式,隶属于集群级别。实例化出特定的对象之后,它会在 API 上注册生成 GVR 类型 URL 端点,并能够作为一种资源类型被使用并实例化相应的对象。自定义资源类型之前,选定其使用的 API 群租名称、版本及新建的资源类型名称,根据这些信息即可创建自定义资源类型,并创建自定义类型的资源对象。
CRD 高级特性:
-
- 支持 subresource,比如 status 或者 scale。
- finalizer,在删除自定义资源的同时去做垃圾回收。
- controller,通过 client-go 中的 informer,实现更强大的功能。
- crd 的 validation 可对资源中的字段进行验证,设置默认值。
CustomResourceDefinition 字段定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | [root@k8s-master01- test -2-26 ~] # kubectl explain customresourcedefinition KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 DESCRIPTION: CustomResourceDefinition represents a resource that should be exposed on the API server. Its name MUST be in the format <.spec.name>.<.spec.group>. FIELDS: apiVersion <string> APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https: //git .k8s.io /community/contributors/devel/sig-architecture/api-conventions .md #resources kind <string> Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https: //git .k8s.io /community/contributors/devel/sig-architecture/api-conventions .md #types-kinds metadata <Object> Standard object's metadata More info: https: //git .k8s.io /community/contributors/devel/sig-architecture/api-conventions .md #metadata spec <Object> -required- spec describes how the user wants the resources to appear status <Object> status indicates the actual state of the CustomResourceDefinition [root@k8s-master01- test -2-26 ~] # |
CustomResourceDefinition.spec 字段定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | [root@k8s-master01- test -2-26 ~] # kubectl explain customresourcedefinition.spec KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 RESOURCE: spec <Object> DESCRIPTION: spec describes how the user wants the resources to appear CustomResourceDefinitionSpec describes how a user wants their resource to appear FIELDS: conversion <Object> conversion defines conversion settings for the CRD. group <string> -required- group is the API group of the defined custom resource. The custom resources are served under ` /apis/ <group>/...`. Must match the name of the CustomResourceDefinition ( in the form `<names.plural>.<group>`). names <Object> -required- names specify the resource and kind names for the custom resource. preserveUnknownFields <boolean> preserveUnknownFields indicates that object fields which are not specified in the OpenAPI schema should be preserved when persisting to storage. apiVersion, kind, metadata and known fields inside metadata are always preserved. This field is deprecated in favor of setting `x-preserve-unknown-fields` to true in `spec.versions[*].schema.openAPIV3Schema`. See https: //kubernetes .io /docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/ #pruning-versus-preserving-unknown-fields for details. scope <string> -required- scope indicates whether the defined custom resource is cluster- or namespace-scoped. Allowed values are `Cluster` and `Namespaced`. versions <[]Object> -required- versions is the list of all API versions of the defined custom resource. Version names are used to compute the order in which served versions are listed in API discovery. If the version string is "kube-like" , it will sort above non "kube-like" version strings , which are ordered lexicographically. "Kube-like" versions start with a "v" , then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA > beta > alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10. [root@k8s-master01- test -2-26 ~] # |
customresourcedefinition.spce 常用字段定义如下:
- conversion <Object>: 定义 CRD 的不同版本之间的格式转换方式。有两个子选项: strategy 和 webhook。
- group <string> -required-: 定义了自定义资源所属的 API 组。自定义资源在`/apis/<group>/…`下提供。必须与 CustomResourceDefinition(格式为 `<names.plural>.<group>`)相同。
- names <Object> -required-: 指定自定义资源的资源和种类名称。
- preserveUnknownFields <boolean>: 表示未指定的对象字段。
- scope <string> -required-: 指示定义的自定义资源是 cluster or Namespace-scoped。允许的值为 'Cluster' 和 'Namespaced'。
- versions <[]Object> -required-: 定义自定义资源的所有API版本的列表。
先看看系统中有哪些 CRD 资源,然后我们再自己创建一个 CRD 资源:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | [root@k8s-master01- test -2-26 ~] # kubectl get CustomResourceDefinition NAME CREATED AT accessors.storage.kubesphere.io 2022-07-01T02:20:20Z alertmanagerconfigs.monitoring.coreos.com 2022-07-01T02:23:04Z alertmanagers.monitoring.coreos.com 2022-07-01T02:23:08Z applications.app.k8s.io 2022-07-01T02:19:04Z applications.argoproj.io 2022-07-01T02:22:10Z applications.gitops.kubesphere.io 2022-07-01T02:22:38Z applicationsets.argoproj.io 2022-07-01T02:22:10Z appprojects.argoproj.io 2022-07-01T02:22:11Z argocdextensions.argoproj.io 2022-07-01T02:22:11Z authorizationpolicies.security.istio.io 2022-07-01T02:21:41Z bgpconfigurations.crd.projectcalico.org 2022-07-01T02:12:34Z bgppeers.crd.projectcalico.org 2022-07-01T02:12:34Z blockaffinities.crd.projectcalico.org 2022-07-01T02:12:35Z clusterconfigurations.installer.kubesphere.io 2022-07-01T02:12:51Z clusterdashboards.monitoring.kubesphere.io 2022-07-01T02:28:13Z clusterinformations.crd.projectcalico.org 2022-07-01T02:12:35Z clusters.cluster.kubesphere.io 2022-07-01T02:19:19Z clustertemplates.devops.kubesphere.io 2022-07-01T02:22:36Z configs.notification.kubesphere.io 2022-07-01T02:27:24Z dashboards.monitoring.kubesphere.io 2022-07-01T02:28:13Z destinationrules.networking.istio.io 2022-07-01T02:21:42Z devopsprojects.devops.kubesphere.io 2022-07-01T02:22:37Z envoyfilters.networking.istio.io 2022-07-01T02:21:42Z exporters.events.kubesphere.io 2022-07-01T02:21:33Z federatedrolebindings.iam.kubesphere.io 2022-07-01T02:19:28Z federatedroles.iam.kubesphere.io 2022-07-01T02:19:31Z federatedusers.iam.kubesphere.io 2022-07-01T02:19:33Z felixconfigurations.crd.projectcalico.org 2022-07-01T02:12:35Z filters.logging.kubesphere.io 2022-07-01T02:17:40Z fluentbitconfigs.logging.kubesphere.io 2022-07-01T02:17:41Z fluentbits.logging.kubesphere.io 2022-07-01T02:17:41Z gateways.gateway.kubesphere.io 2022-07-01T02:19:23Z gateways.networking.istio.io 2022-07-01T02:21:42Z gitrepositories.devops.kubesphere.io 2022-07-01T02:22:37Z globalnetworkpolicies.crd.projectcalico.org 2022-07-01T02:12:35Z globalnetworksets.crd.projectcalico.org 2022-07-01T02:12:35Z globalrolebindings.iam.kubesphere.io 2022-07-01T02:19:35Z globalroles.iam.kubesphere.io 2022-07-01T02:19:38Z groupbindings.iam.kubesphere.io 2022-07-01T02:19:41Z groups .iam.kubesphere.io 2022-07-01T02:19:44Z helmapplications.application.kubesphere.io 2022-07-01T02:19:07Z helmapplicationversions.application.kubesphere.io 2022-07-01T02:19:09Z helmcategories.application.kubesphere.io 2022-07-01T02:19:12Z helmreleases.application.kubesphere.io 2022-07-01T02:19:14Z helmrepos.application.kubesphere.io 2022-07-01T02:19:17Z hostendpoints.crd.projectcalico.org 2022-07-01T02:12:35Z inputs.logging.kubesphere.io 2022-07-01T02:17:41Z ipamblocks.crd.projectcalico.org 2022-07-01T02:12:35Z ipamblocks.network.kubesphere.io 2022-07-01T02:20:00Z ipamconfigs.crd.projectcalico.org 2022-07-01T02:12:35Z ipamhandles.crd.projectcalico.org 2022-07-01T02:12:36Z ipamhandles.network.kubesphere.io 2022-07-01T02:20:03Z ippools.crd.projectcalico.org 2022-07-01T02:12:36Z ippools.network.kubesphere.io 2022-07-01T02:20:06Z istiooperators. install .istio.io 2022-07-01T02:21:42Z jaegers.jaegertracing.io 2022-07-01T02:23:06Z kialis.kiali.io 2022-07-01T02:23:39Z kubecontrollersconfigurations.crd.projectcalico.org 2022-07-01T02:12:36Z loginrecords.iam.kubesphere.io 2022-07-01T02:19:46Z namespacenetworkpolicies.network.kubesphere.io 2022-07-01T02:20:09Z networkpolicies.crd.projectcalico.org 2022-07-01T02:12:36Z networksets.crd.projectcalico.org 2022-07-01T02:12:36Z nginxes.gateway.kubesphere.io 2022-07-01T02:19:25Z notificationmanagers.notification.kubesphere.io 2022-07-01T02:27:24Z outputs.logging.kubesphere.io 2022-07-01T02:17:41Z parsers.logging.kubesphere.io 2022-07-01T02:17:41Z peerauthentications.security.istio.io 2022-07-01T02:21:42Z pipelineruns.devops.kubesphere.io 2022-07-01T02:22:37Z pipelines.devops.kubesphere.io 2022-07-01T02:22:37Z podmonitors.monitoring.coreos.com 2022-07-01T02:23:12Z probes.monitoring.coreos.com 2022-07-01T02:23:27Z prometheuses.monitoring.coreos.com 2022-07-01T02:24:56Z prometheusrules.monitoring.coreos.com 2022-07-01T02:24:11Z receivers.notification.kubesphere.io 2022-07-01T02:27:25Z requestauthentications.security.istio.io 2022-07-01T02:21:42Z resourcequotas. quota .kubesphere.io 2022-07-01T02:20:12Z rolebases.iam.kubesphere.io 2022-07-01T02:19:49Z rulers.events.kubesphere.io 2022-07-01T02:21:33Z rules.auditing.kubesphere.io 2022-07-01T02:21:23Z rules.events.kubesphere.io 2022-07-01T02:21:33Z s2ibinaries.devops.kubesphere.io 2022-07-01T02:22:39Z s2ibuilders.devops.kubesphere.io 2022-07-01T02:22:39Z s2ibuildertemplates.devops.kubesphere.io 2022-07-01T02:22:39Z s2iruns.devops.kubesphere.io 2022-07-01T02:22:40Z serviceentries.networking.istio.io 2022-07-01T02:21:43Z servicemonitors.monitoring.coreos.com 2022-07-01T02:24:17Z servicepolicies.servicemesh.kubesphere.io 2022-07-01T02:20:14Z sidecars.networking.istio.io 2022-07-01T02:21:43Z strategies.servicemesh.kubesphere.io 2022-07-01T02:20:18Z telemetries.telemetry.istio.io 2022-07-01T02:21:43Z templates.devops.kubesphere.io 2022-07-01T02:22:37Z thanosrulers.monitoring.coreos.com 2022-07-01T02:24:26Z users .iam.kubesphere.io 2022-07-01T02:19:52Z virtualservices.networking.istio.io 2022-07-01T02:21:43Z volumesnapshotclasses.snapshot.storage.k8s.io 2022-07-01T02:15:14Z volumesnapshotcontents.snapshot.storage.k8s.io 2022-07-01T02:15:14Z volumesnapshots.snapshot.storage.k8s.io 2022-07-01T02:15:14Z webhooks.auditing.kubesphere.io 2022-07-01T02:21:23Z workloadentries.networking.istio.io 2022-07-01T02:21:43Z workloadgroups.networking.istio.io 2022-07-01T02:21:43Z workspacerolebindings.iam.kubesphere.io 2022-07-01T02:19:55Z workspaceroles.iam.kubesphere.io 2022-07-01T02:19:58Z workspaces.tenant.kubesphere.io 2022-07-01T02:20:23Z workspacetemplates.tenant.kubesphere.io 2022-07-01T02:20:25Z [root@k8s-master01- test -2-26 ~] # |
下面的配置清单中定义了一个名为 user.auth.zuoyang.tech 的 CRD 资源对象,它的群组名称为为 auth.zuoyang.tech,仅支持一个版本级别 v1alpha1,复数形式为 usters,隶属于名称空间级别,因此,它的对象在 API 上的 URL 路径前缀为 /apis/auth.zuoyang.tech/v1alpha1/namespaces/NS_NAME/users/:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | apiVersion: apiextensions.k8s.io /v1 kind: CustomResourceDefinition metadata: name: users .auth.zuoyang.tech spec: group: auth.zuoyang.tech names: kind: User plural: users singular: user shortNames: - u scope: Namespaced versions: - served: true storage: true name: v1alpha1 schema: openAPIV3Schema: type : object properties: spec: type : object properties: userID: type : integer minimum: 1 maximum: 65535 groups : type : array items: type : string email: type : string password: type : string format : password required: [ "userID" , "groups" ] |
配置清单中,spec 嵌套使用的字段都能够见名知意,这里需要特别说明的是,metadata.name 字段的值必须等同于 spec 字段中的 "<name.plural>.<group>" 合并起来的形式。
清单中的 CRD 资源创建于 Kubernetes 集群中之后,继而创建一个新的 CustomResourceDefinition 类型的对象,例如,使用下面的命令列出集群上的 CRD 对象时,命令结果将显示出如下资源名称及创建时间状态信息:
1 2 3 | [root@k8s-master01- test -2-26 ~] # kubectl get crd |grep zuoyang users .auth.zuoyang.tech 2022-07-07T07:48:05Z [root@k8s-master01- test -2-26 ~] # |
现在,在 API 群组 auth.zuoyang.tech/v1beta1 中,users 已经是一个名称空间级别的可用资源类型,用不可按需创建出任意数量的 users 类型的对象,下面就是一个资源清单示例,它定义了一个名为 admin 的 users 对象:
1 2 3 4 5 6 7 8 9 10 11 12 | apiVersion: auth.zuoyang.tech /v1alpha1 kind: User metadata: name: admin namespace: default spec: userID: 1 email: 1635230007@qq.com groups : - superusers - adminstrators password: zuoyang.tech |
1 2 | [root@k8s-master01- test -2-26 ~] # kubectl apply -f user-cr-demo.yaml user.auth.zuoyang.tech /admin created |
"kubectl get users" 可以查看自定义资源的状态信息:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | [root@k8s-master01- test -2-26 ~] # kubectl get users -n default NAME AGE admin 2m39s [root@k8s-master01- test -2-26 ~] # kubectl describe users Name: admin Namespace: default Labels: <none> Annotations: <none> API Version: auth.zuoyang.tech /v1alpha1 Kind: User Metadata: Creation Timestamp: 2022-07-07T07:59:20Z Generation: 1 Managed Fields: API Version: auth.zuoyang.tech /v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io /last-applied-configuration : f:spec: .: f:email: f: groups : f:password: f:userID: Manager: kubectl-client-side-apply Operation: Update Time: 2022-07-07T07:59:20Z Resource Version: 390747 UID: e18c3482-cf85-4898-9ad7-340d47d87572 Spec: Email: 1635230007@qq.com Groups: superusers adminstrators Password: zuoyang.tech User ID: 1 Events: <none> [root@k8s-master01- test -2-26 ~] # |
根据 API 对象 GVR 格式的 URL 规范,users 资源的对象 admin 的引用路径为 /apis/auth.zuoyang.tech/v1alpha1/namespaces/default/users/admin,这一点已通过 descreibe 命令予以证实。而要删除自定义的 users 对象 admin,只要使用通用格式的 kubeclt 命令即可,如 "kubectl delete users admin",或者使用陈述式对象配置命令 “kubectl delete -f <filename>”。
1 2 3 4 5 | [root@k8s-master01- test -2-26 ~] # kubectl delete users admin user.auth.zuoyang.tech "admin" deleted [root@k8s-master01- test -2-26 ~] # kubectl get users -n default No resources found in default namespace. [root@k8s-master01- test -2-26 ~] # |
二、定义资源格式验证
除了对操作请求进行身份验证和授权检查之外,对象配置的变动在存储 etcd 之前还需要经由准入控制器的核验,尤其是验证型(validation)控制器会检查传入的对象格式是否符合有效格式,包括是否设定了不符合定义的数据类型值,以及是否违反了字段的限制规则等。
Kubernetes 自 1.9 版本起支持为 CRD 定义验证(validation 字段)机制用以定义 CRD 可用的有效字段等,这在将设计的自定义资源公开时非常重要。自定义资源可使用 OpenAPI 模式声明验证规则,该模式是 JSON 模式的子集。需要注意的是,OpenAPI 架构并不能支持 JSON 架构的所有功能,而 CRD 验证也不能支持 OpenAPI 架构的所有功能,不过对大多数情况来说,它足够使用了。
下面是一个配置清单(就是上面那个yaml)。它分别定义了 userID、groups、email 和 password 字段的数据类型,并指定了 userID 字段的取值范围,以及 password 字段的数据格式,而且还通过 required 指定 userID 和 groups 是必选字段:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | apiVersion: apiextensions.k8s.io /v1 kind: CustomResourceDefinition metadata: name: users .auth.zuoyang.tech spec: group: auth.zuoyang.tech names: kind: User plural: users singular: user shortNames: - u scope: Namespaced versions: - served: true storage: true name: v1alpha1 schema: openAPIV3Schema: type : object properties: spec: type : object properties: userID: type : integer minimum: 1 maximum: 65535 groups : type : array items: type : string email: type : string password: type : string format : password required: [ "userID" , "groups" ] |
限制了每个字段的数据类型之后,验证机制会检查创建或修改操作中提供的每个字段值等是否违反了格式要求,任何的核验失败都会导致写入操作被拒绝。
自 Kubernetes 1.11 版本开始,kubeclt 即使用服器侧对象信息打印机制,这意味着将由 API Server 决定 kubectl get 命令结果会显示哪些字段。在 CRD 资源中,可在 spec.additionalPrinterColumns 中嵌套定义在对象的详细信息中要打印的字段列表。
下面的配置清单片段定义了要为 users 类型的对象显示相关的四个字段,将其合并至前面定义的 CRD 对象 users 配置清单的 spec 字段中,重新应用到集群中并确保其正常生效:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | [root@k8s-master01- test -2-26 ~] # kubectl explain customresourcedefinition.spec.versions.additionalPrinterColumns KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 RESOURCE: additionalPrinterColumns <[]Object> DESCRIPTION: additionalPrinterColumns specifies additional columns returned in Table output. See https: //kubernetes .io /docs/reference/using-api/api-concepts/ #receiving-resources-as-tables for details. If no columns are specified, a single column displaying the age of the custom resource is used. CustomResourceColumnDefinition specifies a column for server side printing. FIELDS: description <string> description is a human readable description of this column. format <string> format is an optional OpenAPI type definition for this column. The 'name' format is applied to the primary identifier column to assist in clients identifying column is the resource name. See https: //github .com /OAI/OpenAPI-Specification/blob/master/versions/2 .0.md #data-types for details. jsonPath <string> -required- jsonPath is a simple JSON path (i.e. with array notation) which is evaluated against each custom resource to produce the value for this column. name <string> -required- name is a human readable name for the column. priority <integer> priority is an integer defining the relative importance of this column compared to others. Lower numbers are considered higher priority. Columns that may be omitted in limited space scenarios should be given a priority greater than 0. type <string> -required- type is an OpenAPI type definition for this column. See https: //github .com /OAI/OpenAPI-Specification/blob/master/versions/2 .0.md #data-types for details. [root@k8s-master01- test -2-26 ~] # |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | apiVersion: apiextensions.k8s.io /v1 kind: CustomResourceDefinition metadata: name: users .auth.zuoyang.tech spec: group: auth.zuoyang.tech names: kind: User plural: users singular: user shortNames: - u scope: Namespaced versions: - served: true storage: true name: v1alpha1 schema: openAPIV3Schema: type : object properties: spec: type : object properties: userID: type : integer minimum: 1 maximum: 65535 groups : type : array items: type : string email: type : string password: type : string format : password required: [ "userID" , "groups" ] additionalPrinterColumns: - name: userID type : integer description: The user ID. jsonPath: .spec.userID - name: groups type : string description: The goups of the user. jsonPath: .spec.goups - name: email type : string description: The email of the user. jsonPath: .spec.email - name: password type : string description: The password of the user account. jsonPath: .spec.password |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | [root@k8s-master01- test -2-26 ~] # kubectl get crd users.auth.zuoyang.tech NAME CREATED AT users .auth.zuoyang.tech 2022-07-07T08:54:41Z [root@k8s-master01- test -2-26 ~] # kubectl describe crd users.auth.zuoyang.tech Name: users .auth.zuoyang.tech Namespace: Labels: <none> Annotations: <none> API Version: apiextensions.k8s.io /v1 Kind: CustomResourceDefinition Metadata: Creation Timestamp: 2022-07-07T08:54:41Z Generation: 1 Managed Fields: API Version: apiextensions.k8s.io /v1 Fields Type: FieldsV1 fieldsV1: f:status: f:acceptedNames: f:kind: f:listKind: f:plural: f:shortNames: f:singular: f:conditions: k:{ "type" : "Established" }: .: f:lastTransitionTime: f:message: f:reason: f:status: f: type : k:{ "type" : "NamesAccepted" }: .: f:lastTransitionTime: f:message: f:reason: f:status: f: type : Manager: kube-apiserver Operation: Update Subresource: status Time: 2022-07-07T08:54:41Z API Version: apiextensions.k8s.io /v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io /last-applied-configuration : f:spec: f:conversion: .: f:strategy: f:group: f:names: f:kind: f:listKind: f:plural: f:shortNames: f:singular: f:scope: f:versions: Manager: kubectl-client-side-apply Operation: Update Time: 2022-07-07T08:54:41Z Resource Version: 405679 UID: c98f0ba2-7fb4-4e0f-85f4-a55173b5aa64 Spec: Conversion: Strategy: None Group: auth.zuoyang.tech Names: Kind: User List Kind: UserList Plural: users Short Names: u Singular: user Scope: Namespaced Versions: Additional Printer Columns: Description: The user ID. Json Path: .spec.userID Name: userID Type: integer Description: The goups of the user. Json Path: .spec.goups Name: groups Type: string Description: The email of the user. Json Path: .spec.email Name: email Type: string Description: The password of the user account. Json Path: .spec.password Name: password Type: string Name: v1alpha1 Schema: openAPIV3Schema: Properties: Spec: Properties: Email: Type: string Groups: Items: Type: string Type: array Password: Format: password Type: string User ID: Maximum: 65535 Minimum: 1 Type: integer Required: userID groups Type: object Type: object Served: true Storage: true Status: Accepted Names: Kind: User List Kind: UserList Plural: users Short Names: u Singular: user Conditions: Last Transition Time: 2022-07-07T08:54:41Z Message: no conflicts found Reason: NoConflicts Status: True Type: NamesAccepted Last Transition Time: 2022-07-07T08:54:41Z Message: the initial names have been accepted Reason: InitialNamesAccepted Status: True Type: Established Stored Versions: v1alpha1 Events: <none> [root@k8s-master01- test -2-26 ~] # |
三、子资源
前面自定义资源对象 admin 在其详细状态信息输出中没有类似核心资源的 status 字段,该字段是一种用于保存对象当前状态的子资源。
在 Kubernetes 系统的声明式 API 中,status 字段至关重要,它由 Kubernetes 系统自行维护,相关的控制器循环中持续与 API Server 进行通信,并负责确保 status 字段中的状态匹配 spec 字段中定义的期望状态。
在 CRD 中为自定义资源启用 status 字段的方式非常简单,只需要为其定义 spec.subresources.status 字段即可,其内嵌的字段等由系统自行维护,用户无须提供任何额外的配置。它的使用格式如下:
crd.spec.versions.subresources 字段定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | [root@k8s-master01- test -2-26 ~] # kubectl explain crd.spec.versions.subresources KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 RESOURCE: subresources <Object> DESCRIPTION: subresources specify what subresources this version of the defined custom resource have. CustomResourceSubresources defines the status and scale subresources for CustomResources. FIELDS: scale <Object> scale indicates the custom resource should serve a ` /scale ` subresource that returns an `autoscaling /v1 ` Scale object. status <map[string]> status indicates the custom resource should serve a ` /status ` subresource. When enabled: 1. requests to the custom resource primary endpoint ignore changes to the `status` stanza of the object. 2. requests to the custom resource ` /status ` subresource ignore changes to anything other than the `status` stanza of the object. [root@k8s-master01- test -2-26 ~] # |
crd.spec.versions.subresources.scale 字段定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | [root@k8s-master01- test -2-26 ~] # kubectl explain crd.spec.versions.subresources.scale KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 RESOURCE: scale <Object> DESCRIPTION: scale indicates the custom resource should serve a ` /scale ` subresource that returns an `autoscaling /v1 ` Scale object. CustomResourceSubresourceScale defines how to serve the scale subresource for CustomResources. FIELDS: labelSelectorPath <string> labelSelectorPath defines the JSON path inside of a custom resource that corresponds to Scale `status.selector`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.status` or `.spec`. Must be set to work with HorizontalPodAutoscaler. The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form. More info: https: //kubernetes .io /docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions #scale-subresource If there is no value under the given path in the custom resource, the `status.selector` value in the ` /scale ` subresource will default to the empty string. specReplicasPath <string> -required- specReplicasPath defines the JSON path inside of a custom resource that corresponds to Scale `spec.replicas`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.spec`. If there is no value under the given path in the custom resource, the ` /scale ` subresource will return an error on GET. statusReplicasPath <string> -required- statusReplicasPath defines the JSON path inside of a custom resource that corresponds to Scale `status.replicas`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.status`. If there is no value under the given path in the custom resource, the `status.replicas` value in the ` /scale ` subresource will default to 0. [root@k8s-master01- test -2-26 ~] # |
crd.spec.versions.subresources.status字段定义如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 DESCRIPTION: status indicates the custom resource should serve a ` /status ` subresource. When enabled: 1. requests to the custom resource primary endpoint ignore changes to the `status` stanza of the object. 2. requests to the custom resource ` /status ` subresource ignore changes to anything other than the `status` stanza of the object. CustomResourceSubresourceStatus defines how to serve the status subresource for CustomResources. Status is represented by the `.status` JSON path inside of a CustomResource. When set , * exposes a /status subresource for the custom resource * PUT requests to the /status subresource take a custom resource object, and ignore changes to anything except the status stanza * PUT /POST/PATCH requests to the custom resource ignore changes to the status stanza CustomResourceSubresourceStatus defines how to serve the status subresource for CustomResources. Status is represented by the `.status` JSON path inside of a CustomResource. When set , * exposes a /status subresource for the custom resource * PUT requests to the /status subresource take a custom resource object, and ignore changes to anything except the status stanza * PUT /POST/PATCH requests to the custom resource ignore changes to the status stanza [root@k8s-master01- test -2-26 ~] # |
将前面配置清单中 CRD 对象 users 的配置中,再改造下,完成活动对象的修改即可在其实例化出的对象上通过 /status 获取状态信息,例如,对象 admin 的状态引用路径为 /apis/auth.zuoyang.tech/v1alpha1/namespaces/default/users/admin/status。在没有相应资源控制器的情形下,活动对象状态的非计划内变动既不会实时反映到 status 中,也不会问 spec 定义的期望状态转换。
事实上,如果有相应的资源控制器维护自定义的资源类型时,还可以配置自定义资源对象时使用 scale 子资源和 status 子资源协同实现类似 Deployment 或 StatefulSet 等控制器一样对象规模的伸缩功能。例子如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | apiVersion: apiextensions.k8s.io /v1 kind: CustomResourceDefinition metadata: name: users .auth.zuoyang.tech spec: group: auth.zuoyang.tech names: kind: User plural: users singular: user shortNames: - u scope: Namespaced versions: - served: true storage: true name: v1alpha1 schema: openAPIV3Schema: type : object properties: spec: type : object properties: userID: type : integer minimum: 1 maximum: 65535 groups : type : array items: type : string email: type : string password: type : string format : password required: [ "userID" , "groups" ] additionalPrinterColumns: - name: userID type : integer description: The user ID. jsonPath: .spec.userID - name: groups type : string description: The goups of the user. jsonPath: .spec.goups - name: email type : string description: The email of the user. jsonPath: .spec.email - name: password type : string description: The password of the user account. jsonPath: .spec.password subresources: status: {} scale: labelSelectorPath: .status.labelSelector specReplicasPath: .spec.replicas statusReplicasPath: .status.replicas |
crd.spec.versions.subresources.scale 的各内嵌字段介绍如下:
- specReplicasPath <string> -required-: 引用定义在 spec 中用于表示期望的副本数量的字段 jsonPath 格式,如 .spec.relicas。
- statusReplicasPath <string> -required-: 引用保存在 status 中用于表示对象的当前副本数量的字段,jsonPath 格式,如 .status.replicas。
- labelSelectorPath <string>: 可选字段,但若要与 HPA 结合使用则是必选字段,用于引用 status 中用于表示使用的标签选择器字段,jsonPath 格式,如 .status.labelSelector。
为某 CRD 资源定义 scale 子资源之后,即可实例化出支持规模伸缩的自定义资源对象。但必须存在一个相应的资源控制器来持续维护相应的自定义资源对象,以确保其当前状态匹配期望的状态。满足条件后,类似于 Deployment 资源对象等,使用 "kubectl scale" 命令就能完成对自定义资源对象规模的手动伸缩。
四、使用资源类别
类别(categories)[英[ˈkætɪgəriz]美[ˈkætəˌgɔriz]] 是 Kubernetes 1.10 版本引入的一种分组组织自定义资源的方法,定义 CRD 对象时为其指定一个或多个类别,可以通过 kubectl get <category-name> 命令列出该类别中的所有自定义资源对象,all 就是一个常用的内建资源类别。
例如,为前面定义的 CRD 对象 users 的 spec.names 字段中额外内嵌如下配置,便能使得 users 资源类型下的所有对象都隶属于 all 类别:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | [root@k8s-master01- test -2-26 ~] # kubectl explain crd.spec.names KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 RESOURCE: names <Object> DESCRIPTION: names specify the resource and kind names for the custom resource. CustomResourceDefinitionNames indicates the names to serve this CustomResourceDefinition FIELDS: categories <[]string> categories is a list of grouped resources this custom resource belongs to (e.g. 'all' ). This is published in API discovery documents, and used by clients to support invocations like `kubectl get all`. kind <string> -required- kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the `kind` attribute in API calls. listKind <string> listKind is the serialized kind of the list for this resource. Defaults to "`kind`List" . plural <string> -required- plural is the plural name of the resource to serve. The custom resources are served under ` /apis/ <group>/<version>/.../<plural>`. Must match the name of the CustomResourceDefinition ( in the form `<names.plural>.<group>`). Must be all lowercase. shortNames <[]string> shortNames are short names for the resource, exposed in API discovery documents, and used by clients to support invocations like `kubectl get <shortname>`. It must be all lowercase. singular <string> singular is the singular name of the resource. It must be all lowercase. Defaults to lowercased `kind`. [root@k8s-master01- test -2-26 ~] # |
explain crd.spec.names.categories 内嵌字段定义如下:
1 2 3 4 5 6 7 8 9 10 11 | [root@k8s-master01- test -2-26 ~] # kubectl explain crd.spec.names.categories KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io /v1 FIELD: categories <[]string> DESCRIPTION: categories is a list of grouped resources this custom resource belongs to (e.g. 'all' ). This is published in API discovery documents, and used by clients to support invocations like `kubectl get all`. [root@k8s-master01- test -2-26 ~] # |
五、多版本支持
Kubernetes 原生资源的一个引人瞩目的特性是它们能够在 API 版本之间自动和透明地迁移。资源的使用者可以使用混合 API 版本,并且都能够获得他们所期望的资源版本。而 Kubernetes 1.11 版本开始,CRD 支持多个版本,但它们之间的转换必须手动完成。
在 CRD 上使用多版本机制时,将 spec.version 字段替换为 spec.versions 字段,并将支持的各版本以对象列表的形式给出定义即可。不过,多版本并存时,仅其中一个版本且必须有一个版本应标记为存储(customresourcedefinition.spec.versions.storage 字段)版本。
下面的配置清单片断中定义了两个 API 版本,其中仅 v1alpha1 标记为了 storage:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | apiVersion: apiextensions.k8s.io /v1 kind: CustomResourceDefinition metadata: name: users .auth.zuoyang.tech spec: group: auth.zuoyang.tech names: kind: User plural: users singular: user shortNames: - u scope: Namespaced versions: - served: true storage: true name: v1alpha1 schema: openAPIV3Schema: type : object properties: spec: type : object properties: userID: type : integer minimum: 1 maximum: 65535 groups : type : array items: type : string email: type : string password: type : string format : password required: [ "userID" , "groups" ] - served: true storage: false name: v1alpha2 schema: openAPIV3Schema: type : object properties: spec: type : object properties: userID: type : integer minimum: 1 maximum: 65535 groups : type : array items: type : string email: type : string password: type : string format : password required: [ "userID" , "groups" ] additionalPrinterColumns: - name: userID type : integer description: The user ID. jsonPath: .spec.userID - name: groups type : string description: The goups of the user. jsonPath: .spec.goups - name: email type : string description: The email of the user. jsonPath: .spec.email - name: password type : string description: The password of the user account. jsonPath: .spec.password subresources: status: {} scale: labelSelectorPath: .status.labelSelector specReplicasPath: .spec.replicas statusReplicasPath: .status.replicas |
六、自定义控制器基础
仅借助于 CRD 完成资源自定义本身并不能为用户带来太多的价值,它只是资源类型的定义,只是提供了 JSON 格式的数据范式及存储相关数据的能力,至于如何执行数据相关的业务逻辑,时刻确保将 status 中的状态移向 spec 中的状态则是由封装于控制器中的代码来负责实现的。
相应的,为自定义资源类型提供业务逻辑代码的控制器需要由用户自行开发,并运行于 API Server 的客户端程序(通常是托管运行于 kubernetes 系统之上,类似于 Ingress 控制器),这就是所谓的自定义控制器(Custom Controller)。
换句话讲,某特定的 CRD 资源的相关对象发生变动时,如何确保它的当前状态不断地接近期望的状态并非 API Server 的功能,而是相关的专用控制器组件。
事实上,自定义控制器不仅仅能够用于管理 CRD 资源,用户也完全可以仅针对系统内建的核心类型对象开啊更高级的控制器,这些对象类型包括 Service、Deployment、ConfigMap。不过,对于每个 CRD 的自定义类别来说,至少应该存在一个相关的自定义控制器。
简单的来说,控制器负责持续监视资源变动,根据资源的 spec 及 status 中的信息执行某些操作逻辑,并将执行结果更新至资源的 status 中。自定义控制器同样担负着类似的职责,只不过一个特定的控制器通常仅负责一部分特定的资源类型,并执行专有管理逻辑。
Kubernetes 系统内建了许多控制器,如 NodeController、ServiceController 等,它们打包在一起并作为一个统一守护进程 kube-controller-manager。这些控制器基本上都遵循同一钟模式以完成资源管理操作,该模式通常可描述为三种特性:
-
- 声明式 API:用户自定义期望的状态而非要执行的特定操作,如使用 kubectl apply 命令将请求发送至 API Server 存储于 etcd 中,并由 API Server 做出处理响应。
- 异步:客户端的请求于 API Server 存储完成之后即返回成功信息,而无须等待控制器执行的和解循环结束。
- 水平处理(level-based):同一对象有多次变动事件待处理时,控制器仅需要处理其最新一次的变动。这种根据最新观察结果而非历史变化来和解 status 和 spec 的机制即为水平处理机制。
定义控制器实现自定义资源管理行为的最佳方法同样是遵循此类控制器模式进行程序开发,目前,大部分开发接口都是基于 Golang 语言来实现的,相关代码位于客户端库的 client-go/go/tools/cache 和 client-go/util/workqueue 目录中。
简单的来说,控制器包含两个重要组件,Informer/SharedInformer 和 Workqueue,前者负责监视资源对象当前状态的更改,并将事件发送至后者,而后由处理函数进行处理,如下图:
Informer 主要由 Listwatcher、ResourceEvenetHandler 和 ResyncPeriod 三类函数组件进行构造。
-
- Listwather: 是应用于特定名称空间中特定资源对象的列表函数(listFunc)和监视函数(watchFunc),并结合字段选择器为控制器精确限定关注的资源对象。
- ResourceEvenetHandler: 负责处理资源对象状态改变产生的相关通知,其分别使用 AddFunc、UpdateFunc、DeleteFunc 三个函数完成对象的创建、更新和删除操作。
- ResyncPeriod 定义控制器遍历缓存中的所有项目并触发运行 UpdateFunc 的频率,即和解循环的执行频度。
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 全程不用写代码,我用AI程序员写了一个飞机大战
· DeepSeek 开源周回顾「GitHub 热点速览」
· 记一次.NET内存居高不下排查解决与启示
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布:重大改进与新特性概览!
2021-07-07 云原生监控系统Prometheus——Sprint Boot可视化监控