Kubernetes：使用StatefulSet搭建MySQL集群（一主多从）

背景介绍

当我们用K8s管理MySql容器集群时，由于MySql的集群机制是每个实例有自己的存储空间，通过网络进行数据同步，并且数据库应用属于有状态的应用，故采用Deployment方式会很麻烦。

K8s从1.5版本开始使用StatefulSet来编排有状态应用：

  • Pod会被顺序部署和顺序终结：StatefulSet中的各个 Pod会被顺序地创建出来，每个Pod都有一个唯一的ID，在创建后续 Pod 之前，首先要等前面的 Pod 运行成功并进入到就绪状态。删除会销毁StatefulSet 中的每个 Pod，并且按照创建顺序的反序来执行，只有在成功终结后面一个之后，才会继续下一个删除操作。

  • Pod具有唯一网络名称：Pod具有唯一的名称，而且在重启后会保持不变。通过Headless服务，基于主机名，每个 Pod 都有独立的网络地址，这个网域由一个Headless 服务所控制。这样每个Pod会保持稳定的唯一的域名，使得集群就不会将重新创建出的Pod作为新成员。

  • Pod能有稳定的持久存储：StatefulSet中的每个Pod可以有其自己独立的PersistentVolumeClaim对象。即使Pod被重新调度到其它节点上以后，原有的持久磁盘也会被挂载到该Pod。

  • Pod能被通过Headless服务访问到：客户端可以通过服务的域名连接到任意Pod。

 

使用StatefulSet部署MySql

  •  configmap.yaml

ConfigMap 提供 my.cnf 覆盖，独立控制 MySQL 主服务器和从服务器的配置。主服务器能够将复制日志提供给从服务器，从服务器拒绝任何不是通过复制进行的写操作。

 1 apiVersion: v1
 2 kind: ConfigMap
 3 metadata:
 4   name: mysql
 5   labels:
 6     app: mysql
 7 data:
 8   master.cnf: |
 9     # Apply this config only on the master.
10     [mysqld]
11     log-bin  # 主mysql激活二进制日志
12     #设置时区和字符集
13     default-time-zone='+8:00'
14     character-set-client-handshake=FALSE
15     character-set-server=utf8mb4
16     collation-server=utf8mb4_unicode_ci
17     init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
18   slave.cnf: |
19     # Apply this config only on slaves.
20     [mysqld]
21     super-read-only  # 从mysql上面设置为只读
22     #设置时区和字符集
23     default-time-zone='+8:00'
24     character-set-client-handshake=FALSE
25     character-set-server=utf8mb4
26     collation-server=utf8mb4_unicode_ci
27     init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'

 

[root@localhost k8s_mysql]# kubectl apply -f configmap.yaml 
configmap/mysql created
[root@localhost k8s_mysql]# kubectl get cm
NAME    DATA   AGE
mysql   2      107s

 

  • services.yaml

Headless Service 给 StatefulSet 控制器为集合中每个 Pod 创建的 DNS 条目提供了一个宿主。因为 Headless Service 名为 mysql，所以可以通过在同一 Kubernetes 集群和 namespace 中的任何其他 Pod 内解析 <pod-name>.mysql 来访问 Pod。

请注意，只有读取查询才能使用负载平衡的客户端 Service。因为只有一个 MySQL 主服务器，所以客户端应直接连接到 MySQL 主服务器 Pod (通过其在 Headless Service 中的 DNS 条目)以执行写入操作。

 1 # Headless service for stable DNS entries of StatefulSet members.
 2 apiVersion: v1
 3 kind: Service
 4 metadata:
 5   name: mysql
 6   labels:
 7     app: mysql
 8 spec:
 9   ports:
10   - name: mysql
11     port: 3306
12   clusterIP: None
13   selector:
14     app: mysql
15 ---
16 # Client service for connecting to any MySQL instance for reads.
17 # For writes, you must instead connect to the master: mysql-0.mysql.
18 apiVersion: v1
19 kind: Service
20 metadata:
21   name: mysql-read
22   labels:
23     app: mysql
24 spec:
25   ports:
26   - name: mysql
27     port: 3306
28   selector:
29     app: mysql

 

[root@localhost k8s_mysql]# kubectl apply -f services.yaml
service/mysql created
service/mysql-read created
[root@localhost k8s_mysql]# kubectl get service
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.43.0.1       <none>        443/TCP    7d17h
mysql        ClusterIP   None            <none>        3306/TCP   20s
mysql-read   ClusterIP   10.43.118.155   <none>        3306/TCP   20s

 

  • statefulset.yaml
  1 apiVersion: apps/v1
  2 kind: StatefulSet
  3 metadata:
  4   name: mysql
  5 spec:
  6   selector:
  7     matchLabels:
  8       app: mysql
  9   serviceName: mysql
 10   replicas: 3
 11   template:
 12     metadata:
 13       labels:
 14         app: mysql
 15     spec:
 16       initContainers:
 17       - name: init-mysql
 18         image: mysql:5.7
 19         command:
 20         - bash
 21         - "-c"
 22         - |
 23           set -ex
 24           # Generate mysql server-id from pod ordinal index.
 25           [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
 26           ordinal=${BASH_REMATCH[1]}
 27           echo [mysqld] > /mnt/conf.d/server-id.cnf
 28           # Add an offset to avoid reserved server-id=0 value.
 29           echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
 30           # Copy appropriate conf.d files from config-map to emptyDir.
 31           if [[ $ordinal -eq 0 ]]; then
 32             cp /mnt/config-map/master.cnf /mnt/conf.d/
 33           else
 34             cp /mnt/config-map/slave.cnf /mnt/conf.d/
 35           fi
 36         volumeMounts:
 37         - name: conf
 38           mountPath: /mnt/conf.d
 39         - name: config-map
 40           mountPath: /mnt/config-map
 41       - name: clone-mysql
 42         image: gcr.io/google-samples/xtrabackup:1.0
 43         command:
 44         - bash
 45         - "-c"
 46         - |
 47           set -ex
 48           # Skip the clone if data already exists.
 49           [[ -d /var/lib/mysql/mysql ]] && exit 0
 50           # Skip the clone on master (ordinal index 0).
 51           [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
 52           ordinal=${BASH_REMATCH[1]}
 53           [[ $ordinal -eq 0 ]] && exit 0
 54           # Clone data from previous peer.
 55           ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
 56           # Prepare the backup.
 57           xtrabackup --prepare --target-dir=/var/lib/mysql
 58         volumeMounts:
 59         - name: data
 60           mountPath: /var/lib/mysql
 61           subPath: mysql
 62         - name: conf
 63           mountPath: /etc/mysql/conf.d
 64       containers:
 65       - name: mysql
 66         image: mysql:5.7
 67         env:
 68         - name: MYSQL_ALLOW_EMPTY_PASSWORD
 69           value: "1"
 70         ports:
 71         - name: mysql
 72           containerPort: 3306
 73         volumeMounts:
 74         - name: data
 75           mountPath: /var/lib/mysql
 76           subPath: mysql
 77         - name: conf
 78           mountPath: /etc/mysql/conf.d
 79         resources:
 80           requests:
 81             cpu: 500m
 82             memory: 1Gi
 83         livenessProbe:
 84           exec:
 85             command: ["mysqladmin", "ping"]
 86           initialDelaySeconds: 30
 87           periodSeconds: 10
 88           timeoutSeconds: 5
 89         readinessProbe:
 90           exec:
 91             # Check we can execute queries over TCP (skip-networking is off).
 92             command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
 93           initialDelaySeconds: 5
 94           periodSeconds: 2
 95           timeoutSeconds: 1
 96       - name: xtrabackup
 97         image: gcr.io/google-samples/xtrabackup:1.0
 98         ports:
 99         - name: xtrabackup
100           containerPort: 3307
101         command:
102         - bash
103         - "-c"
104         - |
105           set -ex
106           cd /var/lib/mysql
107 
108           # Determine binlog position of cloned data, if any.
109           if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
110             # XtraBackup already generated a partial "CHANGE MASTER TO" query
111             # because we're cloning from an existing slave. (Need to remove the tailing semicolon!)
112             cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
113             # Ignore xtrabackup_binlog_info in this case (it's useless).
114             rm -f xtrabackup_slave_info xtrabackup_binlog_info
115           elif [[ -f xtrabackup_binlog_info ]]; then
116             # We're cloning directly from master. Parse binlog position.
117             [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
118             rm -f xtrabackup_binlog_info xtrabackup_slave_info
119             echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
120                   MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
121           fi
122 
123           # Check if we need to complete a clone by starting replication.
124           if [[ -f change_master_to.sql.in ]]; then
125             echo "Waiting for mysqld to be ready (accepting connections)"
126             until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
127 
128             echo "Initializing replication from clone position"
129             mysql -h 127.0.0.1 \
130                   -e "$(<change_master_to.sql.in), \
131                           MASTER_HOST='mysql-0.mysql', \
132                           MASTER_USER='root', \
133                           MASTER_PASSWORD='', \
134                           MASTER_CONNECT_RETRY=10; \
135                         START SLAVE;" || exit 1
136             # In case of container restart, attempt this at-most-once.
137             mv change_master_to.sql.in change_master_to.sql.orig
138           fi
139 
140           # Start a server to send backups when requested by peers.
141           exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
142             "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
143         volumeMounts:
144         - name: data
145           mountPath: /var/lib/mysql
146           subPath: mysql
147         - name: conf
148           mountPath: /etc/mysql/conf.d
149         resources:
150           requests:
151             cpu: 100m
152             memory: 100Mi
153       volumes:
154       - name: conf
155         emptyDir: {}
156       - name: config-map
157         configMap:
158           name: mysql
159   volumeClaimTemplates:
160   - metadata:
161       name: data
162     spec:
163       accessModes: ["ReadWriteOnce"]
164       resources:
165         requests:
166           storage: 10Gi

 

1 [root@localhost k8s_mysql]# kubectl apply -f satefulset.yml
2 [root@localhost k8s_mysql]# kubectl get po -o wide
3 NAME      READY   STATUS             RESTARTS   AGE     IP           NODE                    NOMINATED NODE   READINESS GATES
4 mysql-0   2/2     Running            5          13s     10.42.1.73   localhost.localdomain   <none>           <none>
5 mysql-1   2/2     Running            0          25s     10.42.1.74   localhost.localdomain   <none>           <none>
6 mysql-2   1/2     CrashLoopBackOff   8          18m     10.42.1.75   localhost.localdomain   <none>           <none>

ps：由于我只有一台性能不咋地的虚拟机，因此导致其中一个节点不能运行起来，不过能在此说明问题就行。

 

  • 验证
 1 root@mysql-0:/# mysql -h mysql-0.mysql
 2 Welcome to the MySQL monitor.  Commands end with ; or \g.
 3 Your MySQL connection id is 1360
 4 Server version: 8.0.18 MySQL Community Server - GPL
 5 
 6 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
 7 
 8 Oracle is a registered trademark of Oracle Corporation and/or its
 9 affiliates. Other names may be trademarks of their respective
10 owners.
11 
12 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
13 
14 mysql> show databases;
15 +--------------------+
16 | Database           |
17 +--------------------+
18 | information_schema |
19 | mysql              |
20 | performance_schema |
21 | sys                |
22 +--------------------+
23 4 rows in set (0.00 sec)
24 
25 mysql> create database addtest;
26 Query OK, 1 row affected (0.01 sec)
27 
28 mysql> show databases;
29 +--------------------+
30 | Database           |
31 +--------------------+
32 | addtest            |
33 | information_schema |
34 | mysql              |
35 | performance_schema |
36 | sys                |
37 +--------------------+
38 5 rows in set (0.00 sec)
39 
40 mysql> exit
41 Bye
42 root@mysql-0:/# mysql -h mysql-1.mysql
43 Welcome to the MySQL monitor.  Commands end with ; or \g.
44 Your MySQL connection id is 1217
45 Server version: 8.0.18 MySQL Community Server - GPL
46 
47 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
48 
49 Oracle is a registered trademark of Oracle Corporation and/or its
50 affiliates. Other names may be trademarks of their respective
51 owners.
52 
53 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
54 
55 mysql> show databases;
56 +--------------------+
57 | Database           |
58 +--------------------+
59 | addtest            |
60 | information_schema |
61 | mysql              |
62 | performance_schema |
63 | sys                |
64 +--------------------+
65 5 rows in set (0.00 sec)
66 
67 mysql> exit

 

posted on 2020-06-04 17:31  Cooper_73  阅读(6227)  评论(1)  编辑  收藏  举报

导航