StatefulSet控制器
应用程序存在“有状态”和“无状态”两种类别。
无状态服务（Stateless Service）：该服务运行的实例不会在本地存储需要持久化的数据，并且多个实例对于同一个请求响应的结果是完全一致的。RC、Deployment、DaemonSet都是管理无状态的服务，它们所管理的Pod的IP、名字，启停顺序等都是随机的。个体对整体无影响，所有pod都是共用一个数据卷的，例如，之前部署的tomcat就是无状态的服务，tomcat被删除，在启动一个新的tomcat，加入到集群即可，跟tomcat的名字无关。
有状态服务（Stateful Service）：该服务运行的实例需要在本地存储持久化数据。StatefulSet是有状态的集合，管理有状态的服务，它所管理的Pod的名称不能随意变化。数据持久化的目录也是不一样，每一个Pod都有自己独有的数据持久化存储目录。比如MySQL主从、redis集群等。即比如MySQL数据库，现在运行在节点A，那么它的数据就存储在节点A上面的，如果这个时候把该服务迁移到节点B去的话，那么就没有之前的数据了，因为它需要去对应的数据目录里面恢复数据，而此时没有任何数据。
一、StatefulSet概述
ReplicaSet控制器可用来管控无状态应用，例如提供静态内容服务的Web服务器程序等，而对于有状态应用的管控，则是另一项专用控制器任务——StatefulSet
ReplicaSet控制器能够从一个预置的pod模板创建一个或者多个pod资源，除了主机名和IP地址等属性之外，这些pod资源并没有本质上的区别，就连它们的名称也是使用同一种散列模式生成，具有很强的相似性。通常，每一个访问请求都会以与其他请求相隔离的方式被这类应用所处理，不分先后也无须关心它们是否存在关联关系，哪怕它们先后来自于同一个请求者。于是，任何一个pod资源都可以被ReplicaSet控制器重构出的新版本所替代，管理员更多关注的也是它们的群体特征，而无须过于关注任何一个个体。提供静态内容服务的Web服务器程序是这类应用的典型代表之一。
对应的，另一类应用程序在处理客户端请求时，对当前请求的处理需要以前一次或多次的请求为基础进行，新客户端发起的请求则会被其施加专用的标识，以确保其后续的请求可以被识别。电商或社交等一类Web应用站点中的服务程序通常属于此类应用。另外还包含了以更强关联关系处理请求的应用，如RDBMS系统上处于同一个事务中的多个请求连接的相关信息，即“状态”，有时甚至还需要持久保存由请求生成的数据，尤其是存储服务类的应用，运行于kubernetes系统上时需要用到持久存储卷。
若ReplicaSet控制器在pod模板中包含了某PVC的引用，则由它创建的所有pod资源都将共享此存储卷。PVC后端的PV访问模型配置为ReadOnlyMany或ReadWriteMany时，这些pod资源中的容器应用挂载存储卷后也就有了相同的数据集。不过，大多数情况是，一个集群系统的分布式应用中，每个实例都有可能需要存储使用不同的数据集，或者各自拥有其专有的数据副本，例如，分布式文件系统GlusterFS和分布式文档存储MongoDB中的每个实例各自使用专有的数据集，分布式服务框架ZooKeeper以及主从复制集群中的Redis的每个实例各自拥有其专用的数据副本。由于ReplicaSet控制器使用的是同一个模板生成pod资源，显然它无法实现为每个pod资源创建专用的存储卷。自主式pod资源没有自愈能力，而组织多个只负责生成一个pod资源的ReplicaSet控制器则有规模扩展不便的尴尬。
其次，除了要用到专有的持久卷之外，有些集群类的分布式应用实例在运行期间还存在角色上的差异，它们存在单向/双向的基于IP地址或主机名的引用关系，例如主从复制集群中的MySQL从节点对主节点的引用。这类应用实例，每一个都应当作一个独立的个体对待。ReplicaSet对象控制下的pod资源重构后，其名称和IP地址都存在变动的可能性，因此也无法适配此种场景之需。而StatefulSet（有状态副本集）则是专门用来满足此类应用的控制器类型，有其管控的每个pod对象都有着固定的主机名和专有的存储卷，即便被重构后也能保持不变。
1. StatefulSet的特性
StatefulSet是pod资源控制器的一种实现，用于部署和扩展有状态应用的pod资源，确保它们的运行顺序及每个pod资源的唯一性。其与ReplicaSet控制器不同的是，虽然所有的Pod对象都基于同一个spec配置所创建，但是StatefulSet需要为每个pod维持一个唯一且固定的标识符，必要时还要为其创建专有的存储卷。StatefulSet主要适用于那些依赖于下列类型资源的应用程序。
稳定的、唯一的网络标识符
稳定的、持久化的存储
有序的、优雅的部署和缩放
有序的、优雅的删除和终止
有序的、自动滚动更新
一般来说，一个典型、完整可用的StatefulSet通常由三个组件构成：Headless Service，volumeClaimTemplates和StatefulSet。其中，Headless Service：用来定义pod网路标识，生成可解析的DNS记录；volumeClaimTemplates：则基于静态或动态的pv供给方式为pod资源提供专有且固定的存储；StatefulSet：管控pod资源。
对于一个拥有N个副本的StatefulSet来说，其pod对象会被有序创建，顺序依次是{0...N-1}，删除则以相反的顺序进行。不过，kubernetes 1.7及其之后的版本也支持并行管理pod对象策略。pod资源的名称格式为$(statefulset name}-$(ordinal)。
2. Headless service是什么
Headless service不分配clusterIP，headless service可以通过解析service的DNS, 返回所有Pod的dns和ip地址 (statefulSet部署的Pod才有DNS)，普通的service,只能通过解析service的DNS返回service的ClusterIP。
3. 为什么要用headless service
在使用Deployment时，创建的Pod名称是没有顺序的，是随机字符串，在用statefulset管理pod时要求pod名称必须是有序的 ，每一个pod不能被随意取代，pod重建后pod名称还是一样的。因为pod IP是变化的，所以要用Pod名称来识别。pod名称是pod唯一性的标识符，必须持久稳定有效。这时候要用到无头服务，它可以给每个Pod一个唯一的名称。
1）.headless service会为service分配一个域名
<service name>.$<namespace name>.svc.cluster.local
2）.StatefulSet会为关联的Pod保持一个不变的Pod Name
statefulset中Pod的名字格式为$(StatefulSet name)-$(pod序号)
3）.StatefulSet会为关联的Pod分配一个dnsName
$<Pod Name>.$<service name>.$<namespace name>.svc.cluster.local
4. 为什么要用volumeClaimTemplate
对于有状态应用都会用到持久化存储，比如mysql主从，由于主从数据库的数据是不能存放在一个目录下的，每个mysql节点都需要有自己独立的存储空间。而在deployment中创建的存储卷是一个共享的存储卷，多个pod使用同一个存储卷，它们数据是同步的，而statefulset定义中的每一个pod都不能使用同一个存储卷，这就需要使用volumeClainTemplate，当在使用statefulset创建pod时，volumeClainTemplate会自动生成一个PVC，从而请求绑定一个PV，每一个pod都有自己专用的存储卷。Pod、PVC和PV对应的关系图如下：
二、StatefulSet基础应用
1. Statefulset资源清单文件编写说明
#查看定义Statefulset资源需要的字段
[root@k8s-master1 ~]# kubectl explain statefulset
KIND: StatefulSet
VERSION: apps/v1
DESCRIPTION:
StatefulSet represents a set of pods with consistent identities. Identities
are defined as:
- Network: A single stable DNS and hostname.
- Storage: As many VolumeClaims as requested. The StatefulSet guarantees
that a given network identity will always map to the same storage identity.
FIELDS:
apiVersion <string> #定义statefulset资源需要使用的api版本
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string> #定义的资源类型
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object> #元数据
spec <Object> #定义容器相关的信息
Spec defines the desired identities of pods in this set.
status <Object> #状态
Status is the current status of Pods in this StatefulSet. This data may be
out of date by some window of time.
#查看statefulset.spec字段
[root@k8s-master1 ~]# kubectl explain statefulset.spec
KIND: StatefulSet
VERSION: apps/v1
RESOURCE: spec <Object>
DESCRIPTION:
Spec defines the desired identities of pods in this set.
A StatefulSetSpec is the specification of a StatefulSet.
FIELDS:
podManagementPolicy <string> #pod管理策略
podManagementPolicy controls how pods are created during initial scale up,
when replacing pods on nodes, or when scaling down. The default policy is
`OrderedReady`, where pods are created in increasing order (pod-0, then
pod-1, etc) and the controller will wait until each pod is ready before
continuing. When scaling down, the pods are removed in the opposite order.
The alternative policy is `Parallel` which will create pods in parallel to
match the desired scale without waiting, and on scale down will delete all
pods at once.
replicas <integer> #副本数
replicas is the desired number of replicas of the given Template. These are
replicas in the sense that they are instantiations of the same Template,
but individual replicas also have a consistent identity. If unspecified,
defaults to 1.
revisionHistoryLimit <integer> #保留的历史版本
revisionHistoryLimit is the maximum number of revisions that will be
maintained in the StatefulSet's revision history. The revision history
consists of all revisions not represented by a currently applied
StatefulSetSpec version. The default value is 10.
selector <Object> -required- #标签选择器，选择它所关联的pod
selector is a label query over pods that should match the replica count. It
must match the pod template's labels. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
serviceName <string> -required- #headless service的名字
serviceName is the name of the service that governs this StatefulSet. This
service must exist before the StatefulSet, and is responsible for the
network identity of the set. Pods get DNS/hostnames that follow the
pattern: pod-specific-string.serviceName.default.svc.cluster.local where
"pod-specific-string" is managed by the StatefulSet controller.
template <Object> -required- #生成pod的模板
template is the object that describes the pod that will be created if
insufficient replicas are detected. Each pod stamped out by the StatefulSet
will fulfill this Template, but have a unique identity from the rest of the
StatefulSet.
updateStrategy <Object> #更新策略
updateStrategy indicates the StatefulSetUpdateStrategy that will be
employed to update Pods in the StatefulSet when a revision is made to
Template.
volumeClaimTemplates <[]Object> #存储卷申请模板
volumeClaimTemplates is a list of claims that pods are allowed to
reference. The StatefulSet controller is responsible for mapping network
identities to claims in a way that maintains the identity of a pod. Every
claim in this list must have at least one matching (by name) volumeMount in
one container in the template. A claim in this list takes precedence over
any volumes in the template, with the same name.
#查看statefulset的spec.template字段
对于template而言，其内部定义的就是pod，pod模板是一个独立的对象
[root@k8s-master1 ~]# kubectl explain statefulset.spec.template
KIND: StatefulSet
VERSION: apps/v1
RESOURCE: template <Object>
DESCRIPTION:
template is the object that describes the pod that will be created if
insufficient replicas are detected. Each pod stamped out by the StatefulSet
will fulfill this Template, but have a unique identity from the rest of the
StatefulSet.
PodTemplateSpec describes the data a pod should have when created from a
template
FIELDS:
metadata <Object>
Standard object's metadata. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec <Object> #定义容器属性的
Specification of the desired behavior of the pod. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
通过上面可以看到，statefulset资源中有两个spec字段。第一个spec声明的是statefulset定义多少个Pod副本（默认将仅部署1个Pod）、匹配Pod标签的选择器、创建pod的模板、存储卷申请模板，第二个spec是spec.template.spec：主要用于Pod里的容器属性等配置。 .spec.template里的内容是声明Pod对象时要定义的各种属性，所以这部分也叫做PodTemplate（Pod模板）。
还有一个值得注意的地方是：在.spec.selector中定义的标签选择器必须能够匹配到spec.template.metadata.labels里定义的Pod标签，否则Kubernetes将不允许创建statefulset。
2. 创建StatefulSet资源
基于StorageClass及其相关的PV动态供给功能创建一个StatefulSet资源，并验证它的各种特性
1）创建StorageClass
依据之前创建的nfs-client-provisioner创建的供应商，创建存储类，供应商的名称为：example.com/nfs
[root@k8s-master1 statefulset]# vim nfs-web-class.yaml You have new mail in /var/spool/mail/root [root@k8s-master1 statefulset]# cat nfs-web-class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-web-class provisioner: example.com/nfs [root@k8s-master1 statefulset]# kubectl apply -f nfs-web-class.yaml storageclass.storage.k8s.io/nfs-web-class created [root@k8s-master1 statefulset]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-web-class example.com/nfs Delete Immediate false 54s
2）创建statefulset对象
[root@k8s-master1 statefulset]# vim statefulset-demo.yaml
You have new mail in /var/spool/mail/root
[root@k8s-master1 statefulset]# cat statefulset-demo.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx
serviceName: "nginx"
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "nfs-web-class"
resources:
requests:
storage: 1Gi
上面示例中的配置定义在statefulset-demo.yaml文件中。由于statefulset资源依赖于一个事先存在的Headless类型的Service资源，因此，上述清单中首先定义了一个nginx的Headless Service资源，用于为关联到的每个pod资源创建DNS资源记录。接着定义了一个名为web的statefulset资源，它通过pod模板创建了两个pod资源副本，并基于volumeClaimTemplates向nfs-web-class存储类请求动态供给pv，从而为每个pod资源提供大小为1GB的专用存储卷。
事实上，定义statefulset资源时，spec中必须要嵌套的字段为“serviceName”和“template”，用于指定关联的Headless Service和要使用的pod模板，“volumeClaimTemplates” 字段用于为pod资源创建专有存储卷PVC模板。
创建清单中定义的相关资源：
[root@k8s-master1 statefulset]# kubectl apply -f statefulset-demo.yaml service/nginx created statefulset.apps/web created
默认情况下，statefulset控制器以串行的方式创建pod副本，如果想要并行创建和删除pod资源，可以设定.spec.podManagementPolicy字段的值为“Parallel”，默认值为“OrderedReady”。在statefulset资源的相关状态中查看相关资源的就绪信息
[root@k8s-master1 statefulset]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 21 5d17h 10.244.169.167 k8s-node2 <none> <none> web-0 1/1 Running 0 38s 10.244.36.125 k8s-node1 <none> <none> web-1 1/1 Running 0 33s 10.244.36.65 k8s-node1 <none> <none> [root@k8s-master1 statefulset]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 60d nginx ClusterIP None <none> 80/TCP 52s [root@k8s-master1 statefulset]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE www-web-0 Bound pvc-5b58e994-4796-4512-8159-c25c884106a9 1Gi RWO nfs-web-class 63s www-web-1 Bound pvc-c2fe3c79-3d11-4db4-b16d-414902c9dc52 1Gi RWO nfs-web-class 58s [root@k8s-master1 statefulset]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-5b58e994-4796-4512-8159-c25c884106a9 1Gi RWO Delete Bound default/www-web-0 nfs-web-class 89s pvc-c2fe3c79-3d11-4db4-b16d-414902c9dc52 1Gi RWO Delete Bound default/www-web-1 nfs-web-class 84s
通过上面可以看到创建的pod是有序的
3. pod资源标识符即存储卷
由StatefulSet控制器创建的pod资源拥有固定、唯一的标识和专用存储卷，即便重新调度或终止后重建，其名称也依然保持不变，且此前的存储卷机器数据不会丢失。
1）Pod资源的固定标识符
如前所述，由StatefulSet控制器创建的pod对象拥有固定且唯一的标识符，它们基于唯一的索引序号及相关的statefulset对象的名称而生成，格式为“<statefulset name>-<ordinal index>”，如下面命令结果所示，两个pod对象的名称分别为web-0和web-1，其中web为控制器的名称：
[root@k8s-master1 statefulset]# kubectl get pods -o wide -l app=nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-0 1/1 Running 0 42m 10.244.36.125 k8s-node1 <none> <none> web-1 1/1 Running 0 42m 10.244.36.65 k8s-node1 <none> <none> [root@k8s-master1 statefulset]#
pod资源的主机名同其资源名称，因此也是带索引序号的名称格式，如下面的命令结果所示：
[root@k8s-master1 statefulset]# for i in 0 1; do kubectl exec web-$i -- sh -c 'hostname';done web-0 web-1
这些名称标识会由StatefulSet资源相关的Headless Service资源创建为DNS资源记录，其域名格式为$(service_name).$(namespace).svc.cluster.local，其中，“cluster.local”是集群的默认域名。在pod资源创建后，与其相关的DNS资源记录格式为“$(pod_name).$(service_name).$(namepsace).svc.cluster.local”。例如前面创建的两个pod资源的DNS记录为web-0.nginx.default.svc.cluster.local和web-1.web-svc.default.svc.cluster.local。
使用kubectl run运行一个提供nslookup命令的容器的，这个命令来自于dnsutils包，通过其交互式接口测试上述两个名称的解析：
[root@k8s-master1 ~]# kubectl exec -it web-1 -- /bin/bash root@web-1:/# nslookup web-0.nginx.default.svc.cluster.local bash: nslookup: command not found root@web-1:/# apt-get update Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB] Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB] Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB] Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8184 kB] Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [186 kB] Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [6344 B] Fetched 8585 kB in 5s (1697 kB/s) Reading package lists... Done root@web-1:/# apt-get install dnsutils -y Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: bind9-dnsutils bind9-host bind9-libs libedit2 libfstrm0 libjson-c5 liblmdb0 libmaxminddb0 libprotobuf-c1 libuv1 Suggested packages: mmdb-bin The following NEW packages will be installed: bind9-dnsutils bind9-host bind9-libs dnsutils libedit2 libfstrm0 libjson-c5 liblmdb0 libmaxminddb0 libprotobuf-c1 libuv1 0 upgraded, 11 newly installed, 0 to remove and 16 not upgraded. Need to get 2775 kB of archives. After this operation, 6199 kB of additional disk space will be used. Get:1 http://deb.debian.org/debian bullseye/main amd64 libuv1 amd64 1.40.0-2 [132 kB] Get:2 http://deb.debian.org/debian bullseye/main amd64 libfstrm0 amd64 0.6.0-1+b1 [21.5 kB] Get:3 http://deb.debian.org/debian bullseye/main amd64 libjson-c5 amd64 0.15-2 [42.8 kB] Get:4 http://deb.debian.org/debian bullseye/main amd64 liblmdb0 amd64 0.9.24-1 [45.0 kB] Get:5 http://deb.debian.org/debian bullseye/main amd64 libmaxminddb0 amd64 1.5.2-1 [29.8 kB] Get:6 http://deb.debian.org/debian bullseye/main amd64 libprotobuf-c1 amd64 1.3.3-1+b2 [27.0 kB] Get:7 http://deb.debian.org/debian-security bullseye-security/main amd64 bind9-libs amd64 1:9.16.33-1~deb11u1 [1410 kB] Get:8 http://deb.debian.org/debian-security bullseye-security/main amd64 bind9-host amd64 1:9.16.33-1~deb11u1 [306 kB] Get:9 http://deb.debian.org/debian bullseye/main amd64 libedit2 amd64 3.1-20191231-2+b1 [96.7 kB] Get:10 http://deb.debian.org/debian-security bullseye-security/main amd64 bind9-dnsutils amd64 1:9.16.33-1~deb11u1 [400 kB] Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 dnsutils all 1:9.16.33-1~deb11u1 [265 kB] Fetched 2775 kB in 2s (1139 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package libuv1:amd64. (Reading database ... 7823 files and directories currently installed.) Preparing to unpack .../00-libuv1_1.40.0-2_amd64.deb ... Unpacking libuv1:amd64 (1.40.0-2) ... Selecting previously unselected package libfstrm0:amd64. Preparing to unpack .../01-libfstrm0_0.6.0-1+b1_amd64.deb ... Unpacking libfstrm0:amd64 (0.6.0-1+b1) ... Selecting previously unselected package libjson-c5:amd64. Preparing to unpack .../02-libjson-c5_0.15-2_amd64.deb ... Unpacking libjson-c5:amd64 (0.15-2) ... Selecting previously unselected package liblmdb0:amd64. Preparing to unpack .../03-liblmdb0_0.9.24-1_amd64.deb ... Unpacking liblmdb0:amd64 (0.9.24-1) ... Selecting previously unselected package libmaxminddb0:amd64. Preparing to unpack .../04-libmaxminddb0_1.5.2-1_amd64.deb ... Unpacking libmaxminddb0:amd64 (1.5.2-1) ... Selecting previously unselected package libprotobuf-c1:amd64. Preparing to unpack .../05-libprotobuf-c1_1.3.3-1+b2_amd64.deb ... Unpacking libprotobuf-c1:amd64 (1.3.3-1+b2) ... Selecting previously unselected package bind9-libs:amd64. Preparing to unpack .../06-bind9-libs_1%3a9.16.33-1~deb11u1_amd64.deb ... Unpacking bind9-libs:amd64 (1:9.16.33-1~deb11u1) ... Selecting previously unselected package bind9-host. Preparing to unpack .../07-bind9-host_1%3a9.16.33-1~deb11u1_amd64.deb ... Unpacking bind9-host (1:9.16.33-1~deb11u1) ... Selecting previously unselected package libedit2:amd64. Preparing to unpack .../08-libedit2_3.1-20191231-2+b1_amd64.deb ... Unpacking libedit2:amd64 (3.1-20191231-2+b1) ... Selecting previously unselected package bind9-dnsutils. Preparing to unpack .../09-bind9-dnsutils_1%3a9.16.33-1~deb11u1_amd64.deb ... Unpacking bind9-dnsutils (1:9.16.33-1~deb11u1) ... Selecting previously unselected package dnsutils. Preparing to unpack .../10-dnsutils_1%3a9.16.33-1~deb11u1_all.deb ... Unpacking dnsutils (1:9.16.33-1~deb11u1) ... Setting up liblmdb0:amd64 (0.9.24-1) ... Setting up libmaxminddb0:amd64 (1.5.2-1) ... Setting up libfstrm0:amd64 (0.6.0-1+b1) ... Setting up libedit2:amd64 (3.1-20191231-2+b1) ... Setting up libprotobuf-c1:amd64 (1.3.3-1+b2) ... Setting up libuv1:amd64 (1.40.0-2) ... Setting up libjson-c5:amd64 (0.15-2) ... Setting up bind9-libs:amd64 (1:9.16.33-1~deb11u1) ... Setting up bind9-host (1:9.16.33-1~deb11u1) ... Setting up bind9-dnsutils (1:9.16.33-1~deb11u1) ... Setting up dnsutils (1:9.16.33-1~deb11u1) ... Processing triggers for libc-bin (2.31-13+deb11u3) ... root@web-1:/# nslookup web-1.nginx.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: web-1.nginx.default.svc.cluster.local Address: 10.244.36.65 #解析的是pod的ip地址 root@web-1:/# nslookup web-0.nginx.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: web-0.nginx.default.svc.cluster.local Address: 10.244.36.125 #解析的是pod的ip地址 root@web-1:/# nslookup nginx #查询service dns，会把对应的pod ip解析出来 Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx.default.svc.cluster.local Address: 10.244.36.125 Name: nginx.default.svc.cluster.local Address: 10.244.36.65 root@web-1:/# dig -t A nginx.default.svc.cluster.local @10.96.0.10 ; <<>> DiG 9.16.33-Debian <<>> -t A nginx.default.svc.cluster.local @10.96.0.10 ;; global options: +cmd ;; Got answer: ;; WARNING: .local is reserved for Multicast DNS ;; You are currently testing what happens when an mDNS query is leaked to DNS ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28040 ;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 0c7029a8c71211f7 (echoed) ;; QUESTION SECTION: ;nginx.default.svc.cluster.local. IN A ;; ANSWER SECTION: nginx.default.svc.cluster.local. 30 IN A 10.244.36.65 nginx.default.svc.cluster.local. 30 IN A 10.244.36.125 ;; Query time: 1 msec ;; SERVER: 10.96.0.10#53(10.96.0.10) ;; WHEN: Sat Oct 01 09:50:56 UTC 2022 ;; MSG SIZE rcvd: 166 root@web-1:/#
Headless Service 资源借助于SRV记录来引用真正提供服务的后端pod资源的主机名称，进行指向包含pod ip地址的记录条目。此外，由statefulset控制器管控的pod资源终止后会由控制器自动进行创建，虽然其IP地址存在变化的可能性，但它的名称标识在重建后保持不变。例如，在另一个终端删除pod资源web-1:
[root@k8s-master1 ~]# kubectl delete pods web-1 pod "web-1" deleted #另一个终端监控发现，删除完成后控制器将随之开始重建pod资源，其名称标识符没有发生变化 [root@k8s-master1 statefulset]# kubectl get pods -w NAME READY STATUS RESTARTS AGE nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 22 5d20h web-0 1/1 Running 0 3h7m web-1 1/1 Running 0 3h7m web-1 1/1 Terminating 0 3h7m web-1 1/1 Terminating 0 3h7m web-1 0/1 Terminating 0 3h7m web-1 0/1 Terminating 0 3h7m web-1 0/1 Terminating 0 3h7m web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 0/1 ContainerCreating 0 1s web-1 1/1 Running 0 2s
Pod资源重建完成后，再次进行名称解析测试。由下面的命令结果可知，Pod资源的DNS标识亦未发生变化，但其IP地址会执向重建后的pod资源地址：
[root@k8s-master1 statefulset]# kubectl exec -it web-1 -- /bin/bash root@web-1:/# dpkg -l |grep dnsutils root@web-1:/# apt-get install dnsutils -y Reading package lists... Done Building dependency tree... Done Reading state information... Done E: Unable to locate package dnsutils root@web-1:/# apt-get update -y Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB] Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB] Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB] Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8184 kB] Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [186 kB] Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [6344 B] Fetched 8585 kB in 5s (1663 kB/s) Reading package lists... Done root@web-1:/# apt-get install dnsutils -y Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: bind9-dnsutils bind9-host bind9-libs libedit2 libfstrm0 libjson-c5 liblmdb0 libmaxminddb0 libprotobuf-c1 libuv1 Suggested packages: mmdb-bin The following NEW packages will be installed: bind9-dnsutils bind9-host bind9-libs dnsutils libedit2 libfstrm0 libjson-c5 liblmdb0 libmaxminddb0 libprotobuf-c1 libuv1 0 upgraded, 11 newly installed, 0 to remove and 16 not upgraded. Need to get 2775 kB of archives. After this operation, 6199 kB of additional disk space will be used. Get:1 http://deb.debian.org/debian bullseye/main amd64 libuv1 amd64 1.40.0-2 [132 kB] Get:2 http://deb.debian.org/debian bullseye/main amd64 libfstrm0 amd64 0.6.0-1+b1 [21.5 kB] Get:3 http://deb.debian.org/debian bullseye/main amd64 libjson-c5 amd64 0.15-2 [42.8 kB] Get:4 http://deb.debian.org/debian bullseye/main amd64 liblmdb0 amd64 0.9.24-1 [45.0 kB] Get:5 http://deb.debian.org/debian bullseye/main amd64 libmaxminddb0 amd64 1.5.2-1 [29.8 kB] Get:6 http://deb.debian.org/debian bullseye/main amd64 libprotobuf-c1 amd64 1.3.3-1+b2 [27.0 kB] Get:7 http://deb.debian.org/debian-security bullseye-security/main amd64 bind9-libs amd64 1:9.16.33-1~deb11u1 [1410 kB] Get:8 http://deb.debian.org/debian-security bullseye-security/main amd64 bind9-host amd64 1:9.16.33-1~deb11u1 [306 kB] Get:9 http://deb.debian.org/debian bullseye/main amd64 libedit2 amd64 3.1-20191231-2+b1 [96.7 kB] Get:10 http://deb.debian.org/debian-security bullseye-security/main amd64 bind9-dnsutils amd64 1:9.16.33-1~deb11u1 [400 kB] Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 dnsutils all 1:9.16.33-1~deb11u1 [265 kB] Fetched 2775 kB in 2s (1189 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package libuv1:amd64. (Reading database ... 7823 files and directories currently installed.) Preparing to unpack .../00-libuv1_1.40.0-2_amd64.deb ... Unpacking libuv1:amd64 (1.40.0-2) ... Selecting previously unselected package libfstrm0:amd64. Preparing to unpack .../01-libfstrm0_0.6.0-1+b1_amd64.deb ... Unpacking libfstrm0:amd64 (0.6.0-1+b1) ... Selecting previously unselected package libjson-c5:amd64. Preparing to unpack .../02-libjson-c5_0.15-2_amd64.deb ... Unpacking libjson-c5:amd64 (0.15-2) ... Selecting previously unselected package liblmdb0:amd64. Preparing to unpack .../03-liblmdb0_0.9.24-1_amd64.deb ... Unpacking liblmdb0:amd64 (0.9.24-1) ... Selecting previously unselected package libmaxminddb0:amd64. Preparing to unpack .../04-libmaxminddb0_1.5.2-1_amd64.deb ... Unpacking libmaxminddb0:amd64 (1.5.2-1) ... Selecting previously unselected package libprotobuf-c1:amd64. Preparing to unpack .../05-libprotobuf-c1_1.3.3-1+b2_amd64.deb ... Unpacking libprotobuf-c1:amd64 (1.3.3-1+b2) ... Selecting previously unselected package bind9-libs:amd64. Preparing to unpack .../06-bind9-libs_1%3a9.16.33-1~deb11u1_amd64.deb ... Unpacking bind9-libs:amd64 (1:9.16.33-1~deb11u1) ... Selecting previously unselected package bind9-host. Preparing to unpack .../07-bind9-host_1%3a9.16.33-1~deb11u1_amd64.deb ... Unpacking bind9-host (1:9.16.33-1~deb11u1) ... Selecting previously unselected package libedit2:amd64. Preparing to unpack .../08-libedit2_3.1-20191231-2+b1_amd64.deb ... Unpacking libedit2:amd64 (3.1-20191231-2+b1) ... Selecting previously unselected package bind9-dnsutils. Preparing to unpack .../09-bind9-dnsutils_1%3a9.16.33-1~deb11u1_amd64.deb ... Unpacking bind9-dnsutils (1:9.16.33-1~deb11u1) ... Selecting previously unselected package dnsutils. Preparing to unpack .../10-dnsutils_1%3a9.16.33-1~deb11u1_all.deb ... Unpacking dnsutils (1:9.16.33-1~deb11u1) ... Setting up liblmdb0:amd64 (0.9.24-1) ... Setting up libmaxminddb0:amd64 (1.5.2-1) ... Setting up libfstrm0:amd64 (0.6.0-1+b1) ... Setting up libedit2:amd64 (3.1-20191231-2+b1) ... Setting up libprotobuf-c1:amd64 (1.3.3-1+b2) ... Setting up libuv1:amd64 (1.40.0-2) ... Setting up libjson-c5:amd64 (0.15-2) ... Setting up bind9-libs:amd64 (1:9.16.33-1~deb11u1) ... Setting up bind9-host (1:9.16.33-1~deb11u1) ... Setting up bind9-dnsutils (1:9.16.33-1~deb11u1) ... Setting up dnsutils (1:9.16.33-1~deb11u1) ... Processing triggers for libc-bin (2.31-13+deb11u3) ... root@web-1:/# nslookup web-1.nginx.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: web-1.nginx.default.svc.cluster.local Address: 10.244.36.123 root@web-1:/# nslookup nginx.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx.default.svc.cluster.local Address: 10.244.36.125 Name: nginx.default.svc.cluster.local Address: 10.244.36.123 root@web-1:/# exit exit You have new mail in /var/spool/mail/root [root@k8s-master1 statefulset]# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nfs-client-provisioner-5d65b75f7-2tv4q 1/1 Running 22 5d20h 10.244.169.167 k8s-node2 <none> <none> web-0 1/1 Running 0 3h14m 10.244.36.125 k8s-node1 <none> <none> web-1 1/1 Running 0 5m58s 10.244.36.123 k8s-node1 <none> <none> [root@k8s-master1 statefulset]#
因此，当客户端尝试向statefulset资源的pod成员发出访问请求时，应该针对Headless Service资源的CNAME(nginx.default.svc.cluster.local)记录进行，它指向的SRV记录包含了当前处于就绪状态的pod资源。当然，若在配置pod模板时定义了pod资源的liveness probe 和readiness probe，考虑到名称标识固定不变，也可以让客户端直接向SRV资源记录（web-0.nginx和web-1.nginx）发出请求。
2）Pod资源的专有存储卷
前面的StatefulSet资源示例示例中，控制器通过了volumeClaimTemplates为每个pod副本自动创建并关联一个PVC对象，它们分别绑定了一个动态供给的PV对象：
[root@k8s-master1 statefulset]# kubectl get pvc -l app=nginx NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE www-web-0 Bound pvc-5b58e994-4796-4512-8159-c25c884106a9 1Gi RWO nfs-web-class 7d6h www-web-1 Bound pvc-c2fe3c79-3d11-4db4-b16d-414902c9dc52 1Gi RWO nfs-web-class 7d6h
PVC存储卷由pod资源中的容器挂载到了/usr/share/nginx/html目录，此为容器应用的nginx进程默认的文档根路径。下面通过kubectl exec命令为每个pod资源于此目录中生成一个测试页面，用于存储卷持久性测试：
[root@k8s-master1 statefulset]# for i in 0 1;do kubectl exec web-$i -- sh -c 'echo $(date),Hostname: $(hostname) > /usr/share/nginx/html/index.html';done
接下来基于一个由cirros镜像启动的客户端pod对象进行访问测试，这样便可通过其DNS名称引用每个pod资源：
[root@k8s-master1 statefulset]# kubectl run -it --image cirros client --restart=Never --rm /bin/sh If you don't see a command prompt, try pressing enter. / # curl web-0.nginx Sat Oct 8 13:44:58 UTC 2022,Hostname: web-0 / # curl web-1.nginx Sat Oct 8 13:44:59 UTC 2022,Hostname: web-1 / #
删除StatefulSet控制器的pod资源，其存储卷并不会被删除，除非用户或管理员手动操作移除操作。因此，在另一个终端删除pod资源web-0,经由StatefulSet控制器重建后，它依然会关联到此前的PVC存储卷上，且此前数据依旧可用：
[root@k8s-master1 statefulset]# kubectl delete pods web-0 pod "web-0" deleted [root@k8s-master1 statefulset]#
在另一个终端监控到的删除及重建过程，如下面的命令将其结果所示：
[root@k8s-master1 ~]# kubectl get pods -l app=nginx -w NAME READY STATUS RESTARTS AGE web-0 1/1 Running 1 7d7h web-1 1/1 Running 1 7d3h web-0 1/1 Terminating 1 7d7h web-0 1/1 Terminating 1 7d7h web-0 0/1 Terminating 1 7d7h web-0 0/1 Terminating 1 7d7h web-0 0/1 Terminating 1 7d7h web-0 0/1 Pending 0 0s web-0 0/1 Pending 0 0s web-0 0/1 ContainerCreating 0 0s web-0 0/1 ContainerCreating 0 2s web-0 1/1 Running 0 2s
而后通过测试用的pod资源接口再次对其进行访问测试，由如下内容可知，存储卷时复用此前的那个：
/ # curl web-0.nginx Sat Oct 8 13:44:58 UTC 2022,Hostname: web-0
由此表明，重建的pod资源被重新调度至哪个节点，此前的PVC资源就会被分配至哪个节点，这样就真正实现了数据的持久化。
三、StatefulSet资源扩缩容
StatefulSet资源的扩缩容与Deployment资源相似，即通过修改资源的副本数来改动其目标pod资源数量。对StatefulSet资源来说，kubectl scale和kubectl patch命令均可实现此功能，也可以使用kubectl edit 命令直接修改其副本数，或者在修改配置文件之后，由kubectl apply命令重新声明。
例如，下面的命令即能将web中的pod副本数量扩展至3个：
[root@k8s-master1 statefulset]# kubectl scale statefulset web --replicas=3 statefulset.apps/web scaled
StatefulSet资源的扩展过程与创建过程的pod资源生成策略相同，默认为顺次进行，而且其名称中的序号也将以现有pod资源的最后一个序号向后进行：
[root@k8s-master1 ~]# kubectl get pods -l app=nginx -w NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 9m56s web-1 1/1 Running 1 7d4h web-2 0/1 Pending 0 0s web-2 0/1 Pending 0 0s web-2 0/1 Pending 0 3s web-2 0/1 ContainerCreating 0 3s web-2 0/1 ContainerCreating 0 4s web-2 1/1 Running 0 5s
也可以使用kubectl edit 命令直接编辑控制器实现扩容，将请求提交给了apiserver，实时修改
[root@k8s-master1 ~]# kubectl get pods -l app=nginx -w NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 18m web-1 1/1 Running 1 7d4h web-2 1/1 Running 0 8m52s web-3 0/1 Pending 0 0s web-3 0/1 Pending 0 0s web-3 0/1 Pending 0 2s web-3 0/1 ContainerCreating 0 2s web-3 0/1 ContainerCreating 0 4s web-3 1/1 Running 0 6s [root@k8s-master1 statefulset]# kubectl get pods -l app=nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-0 1/1 Running 0 20m 10.244.36.70 k8s-node1 <none> <none> web-1 1/1 Running 1 7d4h 10.244.36.69 k8s-node1 <none> <none> web-2 1/1 Running 0 9m57s 10.244.36.71 k8s-node1 <none> <none> web-3 1/1 Running 0 15s 10.244.169.169 k8s-node2 <none> <none>
与扩容操作相对，执行缩容操作只需要将其副本数量调低即可，例如，这里可使用kubectl patch命令将StatefulSet资源web的副本数修补为2个：
[root@k8s-master1 statefulset]# kubectl patch sts web -p '{"spec":{"replicas":2}}'
statefulset.apps/web patched
缩减规模时终止pod资源的默认策略也以pod顺序号逆序逐一进行，直到余下的数量满足目标为止：
[root@k8s-master1 ~]# kubectl get pods -l app=nginx -w NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 24m web-1 1/1 Running 1 7d4h web-2 1/1 Running 0 14m web-3 1/1 Running 0 4m42s web-3 1/1 Terminating 0 4m48s web-3 1/1 Terminating 0 4m48s web-3 0/1 Terminating 0 4m49s web-3 0/1 Terminating 0 4m50s web-3 0/1 Terminating 0 4m50s web-2 1/1 Terminating 0 14m web-2 1/1 Terminating 0 14m web-2 0/1 Terminating 0 14m web-2 0/1 Terminating 0 14m web-2 0/1 Terminating 0 14m
另外，终止pod资源后，其存储卷并不会被删除，因此，缩减规模后若再将其扩展回来，那么此前的数据依然可用，且pod资源名称保持不变。
四、StatefulSet资源升级
自kubernetes 1.7版本起，StatefulSet资源支持自动更新机制，其更新策略将由spec.updateStrategy字段定义，默认为rollingUpdate，即滚动更新。另一个可用策略为OnDelete，即删除pod资源重建
以完成更新。StatefulSet资源的更新机制可用于更新pod资源中的容器镜像、标签、注解和系统资源配额等。
1. 滚动更新
滚动更新StatefulSet控制器的pod资源以逆序的形式从其最大索引编号的pod资源逐一进行，它在终止一个pod资源、更新资源并待其就绪后启动更新下一个资源，即索引号比当前号小1的pod资源。对于主从复制类的集群应用来说，这样也能保证起主节点作用的pod资源最后进行更新，确保兼容性。
StatefulSet的默认更新策略为滚动更新，通过“kubectl describe sts NAME”命令中的输出可以获取相关信息，web控制器的输出结果如下：
[root@k8s-master1 statefulset]# kubectl describe sts web
Name: web
Namespace: default
CreationTimestamp: Sat, 01 Oct 2022 14:47:48 +0800
Selector: app=nginx
Labels: <none>
Annotations: <none>
Replicas: 2 desired | 2 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:latest
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html from www (rw)
Volumes: <none>
Volume Claims:
Name: www
StorageClass: nfs-web-class
Labels: <none>
Annotations: <none>
Capacity: 1Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 40m statefulset-controller create Pod web-0 in StatefulSet web successful
Normal SuccessfulCreate 30m statefulset-controller create Claim www-web-2 Pod web-2 in StatefulSet web success
Normal SuccessfulCreate 30m statefulset-controller create Pod web-2 in StatefulSet web successful
Normal SuccessfulCreate 20m statefulset-controller create Claim www-web-3 Pod web-3 in StatefulSet web success
Normal SuccessfulCreate 20m statefulset-controller create Pod web-3 in StatefulSet web successful
Normal SuccessfulDelete 15m statefulset-controller delete Pod web-3 in StatefulSet web successful
Normal SuccessfulDelete 15m statefulset-controller delete Pod web-2 in StatefulSet web successful
更新pod中的容器镜像可以使用“kubectl set image”命令进行，例如，下面的命令可将web控制器下的pod资源镜像版本升级为“janakiramm/myapp:v2”:
[root@k8s-master1 statefulset]# kubectl set image sts web nginx=ikubernetes/myapp:v2 statefulset.apps/web image updated
更新操作过程，可在另一个终端中使用如下命令进行监控，其逆序操作pod资源的过程如下面命令结果所示：
[root@k8s-master1 ~]# kubectl get pods -l app=nginx -w NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 47m web-1 1/1 Running 1 7d4h web-1 1/1 Terminating 1 7d4h web-1 1/1 Terminating 1 7d4h web-1 0/1 Terminating 1 7d4h web-1 0/1 Terminating 1 7d4h web-1 0/1 Terminating 1 7d4h web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 0/1 ContainerCreating 0 2s web-1 1/1 Running 0 2s web-0 1/1 Terminating 0 47m web-0 1/1 Terminating 0 47m web-0 0/1 Terminating 0 47m web-0 0/1 Terminating 0 47m web-0 0/1 Terminating 0 47m web-0 0/1 Pending 0 0s web-0 0/1 Pending 0 0s web-0 0/1 ContainerCreating 0 1s web-0 0/1 ContainerCreating 0 2s web-0 1/1 Running 0 3s
待更新完成后，获取每个pod资源中的镜像文件版本即可验证其升级结果：
[root@k8s-master1 statefulset]# kubectl get pods -l app=nginx -o custom-columns=NAME:metadata.name,IMAGE:spec.containers[0].image NAME IMAGE web-0 ikubernetes/myapp:v2 web-1 ikubernetes/myapp:v2
其次，也可以通过使用“kubectl describe pods web-0” 命令查看镜像信息更新为： ikubernetes/myapp:v2
[root@k8s-master1 statefulset]# kubectl describe pods web-0
Name: web-0
Namespace: default
Priority: 0
Node: k8s-node1/10.0.0.132
Start Time: Sat, 08 Oct 2022 22:41:25 +0800
Labels: app=nginx
controller-revision-hash=web-b4c5cc7c4
statefulset.kubernetes.io/pod-name=web-0
Annotations: cni.projectcalico.org/podIP: 10.244.36.76/32
cni.projectcalico.org/podIPs: 10.244.36.76/32
Status: Running
IP: 10.244.36.76
IPs:
IP: 10.244.36.76
Controlled By: StatefulSet/web
Containers:
nginx:
Container ID: docker://89afcc7f81bb48fc065c9e8b2ed79de851c3c0cad9b59006906b991e153e231c
Image: ikubernetes/myapp:v2
Image ID: docker://sha256:54202d3f0f3593ea1083e1f2073821c8bad41704767756fad94f23304b057774
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 08 Oct 2022 22:41:27 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from www (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5n29f (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
www:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: www-web-0
ReadOnly: false
default-token-5n29f:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5n29f
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24m default-scheduler Successfully assigned default/web-0 to k8s-node1
Normal Pulled 24m kubelet Container image "ikubernetes/myapp:v2" already present on machine
Normal Created 24m kubelet Created container nginx
Normal Started 24m kubelet Started container nginx
通过上面可以看到pod已经使用刚才更新的镜像ikubernetes/myapp:v2了。
另外，也可以使用"kubectl rollout status sts NAME"命令跟踪StatefulSet资源滚动更新过程中的状态信息。
2. 暂存更新操作
当用户需要设定一个更新操作，但又不希望它立即执行时，可将更新操作予以“暂存”，待条件满足后再手动触发其执行更新。StatefulSet资源的分区更新机制能够实现此项功能。在设定更新操作之前，将sts.spec.updateStrategy.rollingUpdate.partition字段值设置为pod资源的副本数量，即比pod资源的最大索引号大1，这就意味着，所有的pod资源都不会处于可直接更新的分区之内，那么与其后设定的更新策略也不会真正执行，直到用户降低分区编号至现有pod资源索引号范围之内。
下面测试滚动更新的暂存更新操作，首先将StatefulSet资源web的滚动更新分区值设定为2：
[root@k8s-master1 statefulset]# kubectl patch sts web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
statefulset.apps/web patched
而后，将web控制器的pod资源镜像版本更新为ikubernetes/myapp:v1：
[root@k8s-master1 statefulset]# kubectl set image sts web nginx=ikubernetes/myapp:v1 statefulset.apps/web image updated
接着检测各pod资源的镜像文件版本信息，可以发现其版本并未发生变化：
[root@k8s-master1 statefulset]# kubectl get pods -l app=nginx -o custom-columns=NAME:metadata.name,IMAGE:spec.containers[0].image NAME IMAGE web-0 ikubernetes/myapp:v2 web-1 ikubernetes/myapp:v2
此时，即便删除某个pod资源，它依然会基于旧的版本镜像进行重建。例如，下面首先删除了pod对象web-1,随后待其重建操作启动后，再获取与其相关的镜像信息，结果依然显示了旧版本：
[root@k8s-master1 statefulset]# kubectl get pods -o wide -l app=nginx NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-0 1/1 Running 0 27m 10.244.36.76 k8s-node1 <none> <none> web-1 1/1 Running 0 27m 10.244.36.73 k8s-node1 <none> <none> You have new mail in /var/spool/mail/root [root@k8s-master1 statefulset]# kubectl delete pods web-1 pod "web-1" deleted [root@k8s-master1 statefulset]# kubectl get pods web-1 -o custom-columns=NAME:metadata.name,IMAGE:spec.containers[0].image NAME IMAGE web-1 ikubernetes/myapp:v2
由此可见，暂存状态的更新操作对所有的pod资源均布产生影响。
3. 金丝雀部署
将处于暂存状态的更新操作的partition定位于pod资源的最大索引号，即可放出一只金丝雀，由其测试第一轮的更新操作，在确认无误后通过修改partition属性的值更新其他的pod对象时一种更为稳妥的更新操作。
将上述暂停的更新StatefulSet控制器web资源的分区号设置为pod资源的最大索引号1，将会触发web-1的更新操作：
[root@k8s-master1 statefulset]# kubectl patch sts web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":1}}}}'
statefulset.apps/web patched
在另一终端中使用如下命令进行监控，可以看出web-1执行了重建操作：
[root@k8s-master1 ~]# kubectl get pods -l app=nginx -w NAME READY STATUS RESTARTS AGE web-0 1/1 Running 0 35m web-1 1/1 Running 0 6m57s web-1 1/1 Terminating 0 7m2s web-1 1/1 Terminating 0 7m2s web-1 0/1 Terminating 0 7m3s web-1 0/1 Terminating 0 7m4s web-1 0/1 Terminating 0 7m5s web-1 0/1 Pending 0 0s web-1 0/1 Pending 0 0s web-1 0/1 ContainerCreating 0 0s web-1 0/1 ContainerCreating 0 1s web-1 1/1 Running 0 2s
重建完成后，检测所有pod资源使用的镜像文件版本，对比下面命令的输出，可以看出其更新操作再次暂停于web-1的更新操作完成之后：
[root@k8s-master1 statefulset]# kubectl get pods -l app=nginx -o custom-columns=NAME:metadata.name,IMAGE:spec.containers[0].image NAME IMAGE web-0 ikubernetes/myapp:v2 web-1 ikubernetes/myapp:v1
此时，位于非更新分区内的其他pod资源仍不会被更新到新的镜像版本，哪怕它们被删除后重建亦是如此。
4. 分段更新
金丝雀安然度过测试阶段之后，用户便可启动后续其他pod资源的更新操作。在待更新的pod资源数量较少的情况下，直接将partition属性值设置为0，它将逆序完成后续所有pod资源的更新。而当待更新的pod资源较多时，也可以将pod资源以线性或指数级增长方式来分段完成更新操作，操作过程无非是分步更新partition属性值。例如，将web控制器的分区号设置为0以完成剩余的pod资源的更新。
[root@k8s-master1 statefulset]# kubectl patch sts web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
statefulset.apps/web patched
待更新完成后，查看pod中容器的镜像版本信息如下：
[root@k8s-master1 statefulset]# kubectl get pods -l app=nginx -o custom-columns=NAME:metadata.name,IMAGE:spec.containers[0].image NAME IMAGE web-0 ikubernetes/myapp:v1 web-1 ikubernetes/myapp:v1
可以看到，已经全部更新完成。
