kubernetes之PV及PVC案例
目录
kubernetes之PV及PVC案例
概念
默认情况下容器中的磁盘文件是非持久化的，对于运行在容器中的应用来说面临两个问题，第一：当容器挂掉kubelet将重启启动它时，文件将会丢失；第二：当Pod中同时运行多个容器，容器之间需要共享文件时，Kubernetes的Volume解决了这两个问题
官方文档
https://kubernetes.io/zh/docs/concepts/storage/
PersistentVolume（PV）是集群中已由管理员配置的一段网络存储，集群中的存储资源就像一个node节点是一个集群资源，PV是诸如卷之类的卷插件，但是具有独立于使用PV的任何单个pod的生命周期， 该API对象捕获存储的实现细节，即NFS，iSCSI或云提供商特定的存储系统，PV是由管理员添加的的一个存储的描述，是一个全局资源即不隶属于任何namespace，包含存储的类型，存储的大小和访问模式等，它的生命周期独立于Pod，例如当使用它的Pod销毁时对PV没有影响。
PersistentVolumeClaim（PVC）是用户存储的请求，它类似于pod，Pod消耗节点资源，PVC消耗存储资源， 就像pod可以请求特定级别的资源（CPU和内存），PVC是namespace中的资源，可以设置特定的空间大小和访问模式。
kubernetes 从1.0版本开始支持PersistentVolume和PersistentVolumeClaim。
PV是对底层网络存储的抽象，即将网络存储定义为一种存储资源，将一个整体的存储资源拆分成多份后给不同的业务使用。
PVC是对PV资源的申请调用，就像POD消费node节点资源一样,pod是通过PVC将数据保存至PV，PV在保存至存储。
PersistentVolume参数
# kubectl explain PersistentVolume
Capacity： #当前PV空间大小，kubectl explain PersistentVolume.spec.capacity
accessModes ：访问模式，#kubectl explain PersistentVolume.spec.accessModes
ReadWriteOnce – PV只能被单个节点以读写权限挂载，RWO
ReadOnlyMany – PV以可以被多个节点挂载但是权限是只读的,ROX
ReadWriteMany – PV可以被多个节点是读写方式挂载使用,RWX
persistentVolumeReclaimPolicy #删除机制即删除存储卷时候，已经创建好的存储卷由以下删除操作：
#kubectl explain PersistentVolume.spec.persistentVolumeReclaimPolicy
Retain – 删除PV后保持原装，最后需要管理员手动删除
Recycle – 空间回收，及删除存储卷上的所有数据(包括目录和隐藏文件),目前仅支持NFS和hostPath
Delete – 自动删除存储卷
volumeMode #卷类型，kubectl explain PersistentVolume.spec.volumeMode
定义存储卷使用的文件系统是块设备还是文件系统，默认为文件系统
mountOptions #附加的挂载选项列表，实现更精细的权限控制
ro #等
官方提供的基于各后端存储创建的PV支持的访问模式
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options
PersistentVolume参数
#kubectl explain PersistentVolumeClaim.
accessModes ：PVC 访问模式，#kubectl explain PersistentVolumeClaim.spec.volumeMode
ReadWriteOnce – PVC只能被单个节点以读写权限挂载，RWO
ReadOnlyMany – PVC以可以被多个节点挂载但是权限是只读的,ROX
ReadWriteMany – PVC可以被多个节点是读写方式挂载使用,RWX
resources： #定义PVC创建存储卷的空间大小
selector： #标签选择器，选择要绑定的PV
matchLabels #匹配标签名称
matchExpressions #基于正则表达式匹配
volumeName #要绑定的PV名称
volumeMode #卷类型
定义PVC使用的文件系统是块设备还是文件系统，默认为文件系统
运行java应用Jenkins
基于java命令，运行java war包或jar包，本次以jenkins.war 包部署方式为例，且要求jenkins的数据保存至外部存储(NFS或者PVC)，其他java应用看实际需求是否需要将数据保存至外部存储
下载Jenkins War包
https://www.jenkins.io/download/
root@master1:/opt/data/dockerfile/web/jevon/jenkins# ls
Dockerfile jenkins-2.222.1.war
镜像目录文件
root@master1:/opt/data/dockerfile/web/jevon/jenkins# tree
.
├── build-command.sh
├── Dockerfile
├── jenkins-2.222.1.war
└── run_jenkins.sh
0 directories, 4 files
Dockerfile
root@master1:/opt/data/dockerfile/web/jevon/jenkins# cat Dockerfile
FROM harbor.linux.com/webimages/jdk-base:v8.212
MAINTAINER Jevonwei jevonran@163.com
RUN mkdir /apps/jenkins -p
ADD jenkins-2.222.1.war /apps/jenkings/
ADD run_jenkins.sh /usr/bin/
EXPOSE 8080
CMD ["/usr/bin/run_jenkins.sh"]
run_jenkins.sh
root@master1:/opt/data/dockerfile/web/jevon/jenkins# cat run_jenkins.sh
#! /bin/bash
cd /apps/jenkins && java -server -Xms1024m -Xmx1024m -Xss512K -jar jenkins-2.222.1.war --webroot=/apps/jenkins/jenkins-data --httpPort=8080
root@master1:/opt/data/dockerfile/web/pub-images/jenkins# chmod a+x run_jenkins.sh
build-command.sh
root@master1:/opt/data/dockerfile/web/jevon/jenkins# cat build-command.sh
#!/bin/bash
docker build -t harbor.linux.com/danran/jenkins:v2.222.1 .
sleep 1
docker push harbor.linux.com/danran/jenkins:v2.222.1
构建镜像
root@master1:/opt/data/dockerfile/web/jevon/jenkins# bash build-command.sh
Sending build context to Docker daemon 66.29MB
Step 1/7 : FROM harbor.linux.com/webimages/jdk-base:v8.212
---> b2a21bae10f4
Step 2/7 : MAINTAINER Jevonwei jevonran@163.com
---> Running in b58233826de9
Removing intermediate container b58233826de9
---> 56998928e055
Step 3/7 : RUN mkdir /apps/jenkins -p
---> Running in 7d61ce262f02
Removing intermediate container 7d61ce262f02
---> f3ac8e57dde0
Step 4/7 : ADD jenkins-2.222.1.war /apps/jenkins/
---> 0ef8fb2eeb92
Step 5/7 : ADD run_jenkins.sh /usr/bin/
---> 29cb2aa6ca79
Step 6/7 : EXPOSE 8080
---> Running in 26d431e39c5a
Removing intermediate container 26d431e39c5a
---> 3a6a89c04f46
Step 7/7 : CMD ["/usr/bin/run_jenkins.sh"]
---> Running in 8f05103b2a30
Removing intermediate container 8f05103b2a30
---> 5e0134545cb2
Successfully built 5e0134545cb2
Successfully tagged harbor.linux.com/danran/jenkins:v2.222.1
The push refers to repository [harbor.linux.com/danran/jenkins]
a8612df0a8b6: Pushed
b9bfd2cf304d: Pushed
cc45b9ed58cf: Pushed
9f96344cfb8b: Mounted from danran/tomcat-app1
519b1d12005c: Mounted from danran/tomcat-app1
1d12abb2850e: Mounted from danran/tomcat-app1
23a3090b0538: Mounted from danran/nginx-web1
45d27d00345a: Mounted from danran/nginx-web1
6f12928590f2: Mounted from danran/nginx-web1
25cbc4e383db: Mounted from danran/tomcat-app1
9dd0e608b4ce: Mounted from danran/tomcat-app1
6d161c41a7f7: Mounted from danran/nginx-web1
034f282942cd: Mounted from danran/nginx-web1
v2.222.1: digest: sha256:bec9583ffc5be0a4b142dba67b6f43bec319a0781012bb172d13b28d34b13e52 size: 3041
验证jenkins镜像
root@master1:~# docker run -it --rm -p 8081:8080 harbor.linux.com/danran/jenkins:v2.222.1
创建PV/PVC
需要两个PVC，一个保存jenkins的数据，一个保存.jenkins的数据
PV使用NFS存储挂载，请提前确保/data/k8sdata/danran NFS共享目录可挂载
root@master1:~# showmount -e 10.203.104.30
Export list for 10.203.104.30:
/data/k8sdata *
PV
jenkins-persistentvolume.yaml
root@master1:/opt/data/yaml/danran/jenkins# cat jenkins-persistentvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-datadir-pv
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/jenkins-data
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-root-datadir-pv
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/zookeeper-root-data
创建PV
root@master1:/opt/data/yaml/danran/jenkins# kubectl apply -f jenkins-persistentvolume.yaml
persistentvolume/jenkins-datadir-pv created
persistentvolume/jenkins-root-datadir-pv created
验证PV
root@master1:/opt/data/yaml/danran/jenkins# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
jenkins-datadir-pv 20Gi RWO Retain Available 43s
jenkins-root-datadir-pv 20Gi RWO Retain Available 43
PVC
jenkins-persistentvolumeclaim.yaml
root@master1:/opt/data/yaml/danran/jenkins# cat jenkins-persistentvolumeclaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-datadir-pvc
namespace: danran
spec:
accessModes:
- ReadWriteOnce
volumeName: jenkins-datadir-pv
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-root-data-pvc
namespace: danran
spec:
accessModes:
- ReadWriteOnce
volumeName: jenkins-root-datadir-pv
resources:
requests:
storage: 10Gi
创建PVC
root@master1:/opt/data/yaml/danran/jenkins# kubectl apply -f jenkins-persistentvolumeclaim.yaml
persistentvolumeclaim/jenkins-datadir-pvc created
persistentvolumeclaim/jenkins-root-data-pvc created
验证PVC
root@master1:/opt/data/yaml/danran/jenkins# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
jenkins-datadir-pv 20Gi RWO Retain Bound danran/jenkins-datadir-pvc 6m13s
jenkins-root-datadir-pv 20Gi RWO Retain Bound danran/jenkins-root-data-pvc 6m13s
root@master1:/opt/data/yaml/danran/jenkins# kubectl get pvc -n danran
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-datadir-pvc Bound jenkins-datadir-pv 20Gi RWO 2m28s
jenkins-root-data-pvc Bound jenkins-root-datadir-pv 20Gi RWO 2m28s
yaml文件
root@master1:/opt/data/yaml/danran/jenkins# cat jenkins.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: danran-jenkins
name: danran-jenkins-deployment
namespace: danran
spec:
replicas: 1
selector:
matchLabels:
app: danran-jenkins
template:
metadata:
labels:
app: danran-jenkins
spec:
containers:
- name: danran-jenkins-container
image: harbor.linux.com/danran/jenkins:v2.222.1
#imagePullPolicy: IfNotPresent
imagePullPolicy: Always
ports:
- containerPort: 8080
protocol: TCP
name: http
volumeMounts:
- mountPath: "/apps/jenkins/jenkins-data/"
name: jenkins-datadir-danran
- mountPath: "/root/.jenkins"
name: jenkins-root-datadir
volumes:
- name: jenkins-datadir-danran
persistentVolumeClaim:
claimName: jenkins-datadir-pvc
- name: jenkins-root-datadir
persistentVolumeClaim:
claimName: jenkins-root-data-pvc
---
kind: Service
apiVersion: v1
metadata:
labels:
app: danran-jenkins
name: danran-nginx-service
namespace: danran
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 38080
selector:
app: danran-jenkins
创建Pod
root@master1:/opt/data/yaml/danran/jenkins# kubectl apply -f jenkins.yaml
deployment.apps/danran-jenkins-deployment created
service/danran-nginx-service created
验证Pod
root@master1:~# kubectl get pod -n danran
NAME READY STATUS RESTARTS AGE
danran-jenkins-deployment-788fd6d64f-n86cz 1/1 Running 0 12s
web 登录测试
在pod中查看AdminPassword
[root@danran-jenkins-deployment-788fd6d64f-n86cz /]# cat /root/.jenkins/secrets/initialAdminPassword
30a2d1a3a59a4d3dbf9e3ab598cfd099
将AdminPassword填入web，登录
实战案例之Redis 服务
http://download.redis.io/releases/redis-4.0.14.tar.gz
构建redis镜像
镜像文件结构
root@master1:/opt/data/dockerfile/web/jevon/redis# tree
.
├── build-command.sh
├── Dockerfile
├── redis-4.0.14.tar.gz
├── redis.conf
└── run_redis.sh
0 directories, 5 files
Dockerfile
root@master1:/opt/data/dockerfile/web/jevon/redis# cat Dockerfile
FROM harbor.linux.com/baseimages/centos-jevon-base:7.7.1908
MAINTAINER JevonWei "jevonran@163.com"
ADD redis-4.0.14.tar.gz /usr/local/src/
RUN ln -sv /usr/local/src/redis-4.0.14 /usr/local/redis && cd /usr/local/redis && make && cp src/redis-cli /usr/sbin/ && cp src/redis-server /usr/sbin/ && mkdir -pv /data/redis-data && mkdir -pv /var/log/redis/
ADD redis.conf /usr/local/redis/redis.conf
ADD run_redis.sh /usr/local/redis/run_redis.sh
EXPOSE 6379
CMD ["/usr/local/redis/run_redis.sh"]
编辑redis.conf
修改redis.conf配置文件的以下选项
root@master1:/opt/data/dockerfile/web/jevon/redis# cat redis.conf
bind 0.0.0.0
save 900 1
save 5 1
save 300 10
dir /data/redis-data
requirepass danran
run_redis.sh
root@master1:/opt/data/dockerfile/web/jevon/redis# cat run_redis.sh
#!/bin/bash
/usr/sbin/redis-server /usr/local/redis/redis.conf
tail -f /etc/hosts
build-command.sh
root@master1:/opt/data/dockerfile/web/jevon/redis# cat build-command.sh
#!/bin/bash
TAG=$1
docker build -t harbor.linux.com/danran/redis:${TAG} .
sleep 1
docker push harbor.linux.com/danran/redis:${TAG}
构建镜像
root@master1:/opt/data/dockerfile/web/jevon/redis# bash build-command.sh v1
运行redis服务
创建PV与PVC
PV
jenkins-persistentvolume.yaml
root@master1:/opt/data/yaml/danran/redis# cat redis-persistentvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-datadir-pv-1
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/redis-datadir
创建PV
root@master1:/opt/data/yaml/danran/redis# kubectl apply -f redis-persistentvolume.yaml
persistentvolume/redis-datadir-pv created
PVC
redis-persistentvolumeclaim.yaml
root@master1:/opt/data/yaml/danran/redis# cat redis-persistentvolumeclaim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-datadir-pvc
namespace: danran
spec:
accessModes:
- ReadWriteOnce
volumeName: redis-datadir-pv-1
resources:
requests:
storage: 10Gi
创建PVC
root@master1:/opt/data/yaml/danran/redis# kubectl apply -f redis-persistentvolumeclaim.yaml
persistentvolumeclaim/redis-datadir-pvc created
验证PV/PVC
root@master1:/opt/data/yaml/danran/redis# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
redis-datadir-pv-1 20Gi RWO Retain Bound danran/redis-datadir-pvc 5m36s
root@master1:/opt/data/yaml/danran/redis# kubectl get pvc -n danran
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
redis-datadir-pvc Bound redis-datadir-pv-1 20Gi RWO 11m
yaml文件
root@master1:/opt/data/yaml/danran/redis# cat redis.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: devops-redis
name: deploy-devops-redis
namespace: danran
spec:
replicas: 1
selector:
matchLabels:
app: devops-redis
template:
metadata:
labels:
app: devops-redis
spec:
containers:
- name: redis-container
image: harbor.linux.com/danran/redis:v1
#imagePullPolicy: IfNotPresent
imagePullPolicy: Always
volumeMounts:
- mountPath: "/data/redis-data"
name: redis-datadir
volumes:
- name: redis-datadir
persistentVolumeClaim:
claimName: redis-datadir-pvc
---
kind: Service
apiVersion: v1
metadata:
labels:
app: devops-redis
name: srv-devops-redis
namespace: danran
spec:
type: NodePort
ports:
- name: http
port: 6379
targetPort: 6379
nodePort: 36379
selector:
app: devops-redis
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
构建pod
root@master1:/opt/data/yaml/danran/redis# kubectl apply -f redis.yaml
deployment.apps/deploy-devops-redis unchanged
service/srv-devops-redis created
查看srv-devops-redis service
root@master1:~# kubectl get svc -n danran
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
danran-nginx-service NodePort 172.28.129.39 <none> 80:38080/TCP 3h25m
danran-nginx-spec NodePort 172.28.111.204 <none> 80:40002/TCP,443:40043/TCP 3d14h
danran-tomcat-app1-service NodePort 172.28.134.178 <none> 80:40004/TCP 2d15h
srv-devops-redis NodePort 172.28.158.139 <none> 6379:36379/TCP 5m26s
外部客户端访问redis
root@master1:~# redis-cli -h 10.203.104.26 -p 36379 -a danran
实战案例之MySQL 主从架构
https://kubernetes.io/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
https://kubernetes.io/zh/docs/tasks/run-application/run-replicated-stateful-application/
https://www.kubernetes.org.cn/statefulset
基于StatefulSet实现
Pod调度运行时，如果应用不需要任何稳定的标示、有序的部署、删除和扩展，则应该使用一组无状态副本的控制器来部署应用，例如 Deployment 或 ReplicaSet更适合无状态服务需求，而StatefulSet适合管理所有有状态的服务，比如MySQL、MongoDB集群等。
基于StatefulSet 实现的MySQL 一主多从架构
StatefulSet本质上是Deployment的一种变体，在v1.9版本中已成为GA版本，它为了解决有状态服务的问题，它所管理的Pod拥有固定的Pod名称，启停顺序，在StatefulSet中，Pod名字称为网络标识(hostname)，还必须要用到共享存储。
在Deployment中，与之对应的服务是service，而在StatefulSet中与之对应的headless service，headlessservice，即无头服务，与service的区别就是它没有Cluster IP，解析它的名称时将返回该Headless Service对应的全部Pod的Endpoint列表。
StatefulSet 特点：
-> 给每个pdo分配固定且唯一的网络标识符
-> 给每个pod分配固定且持久化的外部存储
-> 对pod进行有序的部署和扩展
-> 对pod进有序的删除和终止
-> 对pod进有序的自动滚动更新
StatefulSet的组成部分
Headless Service：用来定义Pod网络标识( DNS domain)。
StatefulSet：定义具体应用，有多少个Pod副本，并为每个Pod定义了一个域名。
volumeClaimTemplates： 存储卷申请模板，创建PVC，指定pvc名称大小，将自动创建pvc，且pvc必须由存储类供应。
镜像准备
https://github.com/docker-library/ #github 下载地址
准备xtrabackup镜像
root@master1:~# docker pull registry.cn-hangzhou.aliyuncs.com/hxpdocker/xtrabackup:1.0
root@master1:~/package# docker tag registry.cn-hangzhou.aliyuncs.com/hxpdocker/xtrabackup:1.0 harbor.linux.com/danran/xtrabackup:1.0
root@master1:~/package# docker push harbor.linux.com/danran/xtrabackup:1.0
准备mysql 镜像
root@master1:~# docker pull mysql:5.7
root@master1:~/package# docker tag mysql:5.7.29 harbor.linux.com/danran/mysql:5.7.29
root@master1:~/package# docker push harbor.linux.com/danran/mysql:5.7.29
创建PV
创建五个PV，每个pod挂载一个pv
mysql-persistentvolume.yaml
root@master1:/opt/data/yaml/danran/mysql# cat mysql-persistentvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-datadir-1
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/mysql-datadir-1
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-datadir-2
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/mysql-datadir-2
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-datadir-3
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/mysql-datadir-3
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-datadir-4
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/mysql-datadir-4
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-datadir-5
namespace: danran
spec:
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/mysql-datadir-5
创建PV
root@master1:/opt/data/yaml/danran/mysql# kubectl apply -f mysql-persistentvolume.yaml
persistentvolume/mysql-datadir-1 created
persistentvolume/mysql-datadir-2 created
persistentvolume/mysql-datadir-3 created
persistentvolume/mysql-datadir-4 created
persistentvolume/mysql-datadir-5 created
root@master1:/opt/data/yaml/danran/mysql# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
mysql-datadir-1 20Gi RWO Retain Available 118s
mysql-datadir-2 20Gi RWO Retain Available 118s
mysql-datadir-3 20Gi RWO Retain Available 118s
mysql-datadir-4 20Gi RWO Retain Available 118s
mysql-datadir-5 20Gi RWO Retain Available 118s
部署Mysql
部署 MySQL 示例，包含一个 ConfigMap，两个 Services，与一个 StatefulSet。
文件结构
root@master1:/opt/data/yaml/danran/mysql# tree
.
├── mysql-configmap.yaml
├── mysql-persistentvolume.yaml
├── mysql-services.yaml
└── mysql-statefulset.yaml
0 directories, 4 files
ConfigMap
root@master1:/opt/data/yaml/danran/mysql# cat mysql-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql
labels:
app: mysql
data:
master.cnf: |
# Apply this config only on the master.
[mysqld]
log-bin
log_bin_trust_function_creators=1
lower_case_table_names=1
slave.cnf: |
# Apply this config only on slaves.
[mysqld]
super-read-only
log_bin_trust_function_creators=1
Services
以下 YAML 配置文件创建服务
root@master1:/opt/data/yaml/danran/mysql# cat mysql-services.yaml
# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
clusterIP: None
selector:
app: mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the master: mysql-0.mysql.
apiVersion: v1
kind: Service
metadata:
name: mysql-read
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
StatefulSet
拷贝https://kubernetes.io/zh/docs/tasks/run-application/run-replicated-stateful-application/的StatefulSet中的配置，并将image修改为harbor的镜像
创建pod
root@master1:/opt/data/yaml/danran/mysql# kubectl apply -f .
configmap/mysql created
service/mysql created
service/mysql-read created
statefulset.apps/mysql created
验证mysql pod
root@master1:/opt/data/yaml/danran/mysql# kubectl get pod -A | grep mysql
default mysql-0 2/2 Running 0 115s
default mysql-1 2/2 Running 1 87s
default mysql-2 2/2 Running 1 59s
NFS中验证mysql数据
三个mysql 随机绑定了三个pv
root@ha1:~# ls /data/k8sdata/danran/mysql-datadir-1/
mysql
root@ha1:~# ls /data/k8sdata/danran/mysql-datadir-2/
mysql
root@ha1:~# ls /data/k8sdata/danran/mysql-datadir-3/
root@ha1:~# ls /data/k8sdata/danran/mysql-datadir-4/
root@ha1:~# ls /data/k8sdata/danran/mysql-datadir-5/
mysql
验证
在mysql-0 master中创建数据库danran
root@master1:~# kubectl exec -it mysql-0 bash
root@mysql-0:/# mysql
mysql> create database danran;
Query OK, 1 row affected (0.00 sec)
在mysql-1中查看新建的danran数据库是否存在
root@master1:~# kubectl exec -it mysql-1 bash
Defaulting container name to mysql.
Use 'kubectl describe pod/mysql-1 -n default' to see all of the containers in this pod.
root@mysql-1:/# mysql
mysql> show databases;
+------------------------+
| Database |
+------------------------+
| information_schema |
| danran |
| mysql |
| performance_schema |
| sys |
| xtrabackup_backupfiles |
+------------------------+
6 rows in set (0.00 sec)
k8s 示例之WordPress
https://cn.wordpress.org/
https://cn.wordpress.org/download/releases/
LNMP案例之基于Nginx+PHP实现WordPress博客站点，要求Nginx+PHP运行在同一个Pod的不同容器，MySQL运行与default的namespace并可以通过service name增删改查数据库
PHP代码通过NFS共享挂载到每个容器的的代码目录中
PHP镜像
官方PHP镜像
root@master1:~# docker pull php:5.6.40-fpm
root@master1:~# docker tag php:5.6.40-fpm harbor.linux.com/danran/php:5.6.40-fpm
root@master1:~# docker push harbor.linux.com/danran/php:5.6.40-fpm
准备PHP镜像
目录结构
root@master1:/opt/data/dockerfile/web/jevon/wordpress/php# tree
.
├── build-command.sh
├── Dockerfile
├── run_php.sh
└── www.conf
0 directories, 4 files
Dockerfile
root@master1:/opt/data/dockerfile/web/jevon/wordpress/php# cat Dockerfile
#PHP Base Image
FROM harbor.linux.com/baseimages/centos-jevon-base:7.7.1908
MAINTAINER jevonran@163.com
RUN yum install -y https://mirrors.tuna.tsinghua.edu.cn/remi/enterprise/remi-release-7.rpm && yum install -y php56-php-fpm php56-php-mysql -y
ADD www.conf /opt/remi/php56/root/etc/php-fpm.d/www.conf
ADD run_php.sh /usr/local/bin/run_php.sh
EXPOSE 9000
CMD ["/usr/local/bin/run_php.sh]
run_php.sh
root@master1:/opt/data/dockerfile/web/jevon/wordpress/php# cat run_php.sh
#!/bin/bash
/opt/remi/php56/root/usr/sbin/php-fpm
taif -f /etc/hosts
root@master1:/opt/data/dockerfile/web/jevon/wordpress/php# chmod +x run_php.sh
build-command.sh
root@master1:/opt/data/dockerfile/web/jevon/wordpress/php# cat build-command.sh
#!/bin/bash
TAG=$1
docker build -t harbor.linux.com/danran/wordpress-php:${TAG} .
sleep 1
docker push harbor.linux.com/danran/wordpress-php:${TAG}
构建镜像
root@master1:/opt/data/dockerfile/web/jevon/wordpress/php# bash build-command.sh v1
准备wordpress Nginx 基础镜像
目录结构
root@master1:/opt/data/dockerfile/web/pub-images/nginx-base-wordpress# tree
.
├── build-command.sh
├── Dockerfile
└── nginx-1.14.2.tar.gz
0 directories, 3 files
Dockerfile
root@master1:/opt/data/dockerfile/web/pub-images/nginx-base-wordpress# cat Dockerfile
#Nginx Base Image
FROM harbor.linux.com/baseimages/centos-jevon-base:7.7.1908
MAINTAINER jevonran@163.com
RUN yum install -y vim wget tree lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlibdevel openssl openssl-devel iproute net-tools iotop
ADD nginx-1.14.2.tar.gz /usr/local/src/
RUN cd /usr/local/src/nginx-1.14.2 && ./configure --prefix=/apps/nginx && make && make install && ln -sv /apps/nginx/sbin/nginx /usr/sbin/nginx && rm -rf /usr/local/src/nginx-1.14.2.tar.gz
build-command.sh
root@master1:/opt/data/dockerfile/web/pub-images/nginx-base-wordpress# cat build-command.sh
#!/bin/bash
docker build -t harbor.linux.com/webimages/nginx-base-wordpress:v1.14.2 .
sleep 1
docker push harbor.linux.com/webimages/nginx-base-wordpress:v1.14.2
构建镜像
root@master1:/opt/data/dockerfile/web/pub-images/nginx-base-wordpress# bash build-command.sh
准备wordpress PHP+Nginx镜像
目录结构
root@master1:/opt/data/dockerfile/web/jevon/wordpress/nginx# tree
.
├── build-command.sh
├── Dockerfile
├── index.html
├── nginx.conf
└── run_nginx.sh
0 directories, 5 files
Dockerfile
root@master1:/opt/data/dockerfile/web/jevon/wordpress/nginx# cat Dockerfile
#Nginx Base Image
FROM harbor.linux.com/webimages/nginx-base-wordpress:v1.14.2
ADD nginx.conf /apps/nginx/conf/nginx.conf
ADD run_nginx.sh /apps/nginx/sbin/run_nginx.sh
RUN mkdir -pv /home/nginx/wordpress
RUN chown nginx:nginx /home/nginx/wordpress/ -R
EXPOSE 80 443
CMD ["/apps/nginx/sbin/run_nginx.sh"]
nginx.conf
root@master1:/opt/data/dockerfile/web/jevon/wordpress/nginx# cat nginx.conf
user nginx nginx;
worker_processes auto;
events {
worker_connections 1024;
# multi_accept on;
}
http {
sendfile on;
keepalive_timeout 65;
client_max_body_size 10M;
client_body_buffer_size 16K;
client_body_temp_path /apps/nginx/tmp/ 1 2 2;
include mime.types;
default_type application/octet-stream;
gzip on;
server {
listen 80;
server_name localhost;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
root /home/nginx/wordpress;
index index.php index.html index.htm;
}
# pass PHP scripts to FastCGI server
#
location ~ \.php$ {
root /home/nginx/wordpress;
fastcgi_index index.php;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
run_nginx.sh
root@master1:/opt/data/dockerfile/web/jevon/wordpress/nginx# cat run_nginx.sh
#!/bin/bash
/apps/nginx/sbin/nginx
tail -f /etc/hosts
build-command.sh
root@master1:/opt/data/dockerfile/web/jevon/wordpress/nginx# cat build-command.sh
#!/bin/bash
TAG=$1
docker build -t harbor.linux.com/danran/wordpress-nginx:${TAG} .
sleep 1
docker push harbor.linux.com/danran/wordpress-nginx:${TAG}
构建nginx镜像
root@master1:/opt/data/dockerfile/web/jevon/wordpress/nginx# bash build-command.sh v1
运行WordPress站点
使用官方PHP镜像运行PHP环境，WordPress页面文件保存在后端存储NFS服务器
yaml
root@master1:/opt/data/yaml/danran/wordpress# cat wordpress.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: wordpress-app
name: wordpress-app-deployment
namespace: danran
spec:
replicas: 1
selector:
matchLabels:
app: wordpress-app
template:
metadata:
labels:
app: wordpress-app
spec:
containers:
- name: wordpress-app-nginx
image: harbor.linux.com/danran/wordpress-nginx:v1
#imagePullPolicy: IfNotPresent
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
name: http
- containerPort: 443
protocol: TCP
name: https
volumeMounts:
- name: wordpress
mountPath: /home/nginx/wordpress
readOnly: false
- name: wordpress-app-php
image: harbor.linux.com/danran/php:5.6.40-fpm
#imagePullPolicy: IfNotPresent
imagePullPolicy: Always
ports:
- containerPort: 9000
protocol: TCP
name: http
volumeMounts:
- name: wordpress
mountPath: /home/nginx/wordpress
readOnly: false
volumes:
- name: wordpress
nfs:
server: 10.203.104.30
path: /data/k8sdata/danran/wordpress
---
kind: Service
apiVersion: v1
metadata:
labels:
app: wordpress-app
name: wordpress-app-spec
namespace: danran
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
nodePort: 30031
- name: https
port: 443
protocol: TCP
targetPort: 443
nodePort: 30443
selector:
app: wordpress-app
运行wordpress
root@master1:/opt/data/yaml/danran/wordpress# kubectl apply -f wordpress.yaml
deployment.apps/wordpress-app-deployment created
service/wordpress-app-spec created
验证pod
root@master1:~# kubectl get pods -n danran
NAME READY STATUS RESTARTS AGE
wordpress-app-deployment-55bb954d5f-hmxn8 2/2 Running 0 33s
在pod中确认nginx及php是否启用，确认nfs共享目录是否挂载
测试Nginx访问
在NFS 共享目录中创建一个index.html测试页，并将该共享目录的权限设置nginx的启动账号nginx可读,Pod中nginx的UID为2020
root@ha1:~# cd /data/k8sdata/danran/
root@ha1:/data/k8sdata/danran# chown 2020.2020 wordpress/ -R
root@ha1:/data/k8sdata/danran# cat wordpress/index.html
Nginx Web Page
浏览器中访问Node节点的nginx暴露端口30031(http://10.203.104.27:30031/index.html)
测试PHP访问
在NFS 共享目录中创建一个PHP测试页，并将该测试页的权限设置nginx的启动账号nginx可读,Pod中nginx的UID为2020
root@ha1:/data/k8sdata/danran# cat wordpress/index.php
<?php
phpinfo()
?>
root@ha1:/data/k8sdata/danran/wordpress# chown 2020.2020 index.php
浏览器中访问php测试页(http://10.203.104.27:30031/index.php)
将WordPress保存至NFS的共享目录
root@ha1:/data/k8sdata/danran/wordpress# ls
wordpress wordpress-5.2.5.tar.gz
root@ha1:/data/k8sdata/danran/wordpress# tar zxvf wordpress-5.2.5.tar.gz
将wordpress解压后的数据移动到/data/k8sdata/danran/wordpress目录下
root@ha1:/data/k8sdata/danran/wordpress# mv wordpress/* .
root@ha1:/data/k8sdata/danran/wordpress# rmdir wordpress/
root@ha1:/data/k8sdata/danran/wordpress# ls
index.php readme.html wp-activate.php wp-blog-header.php wp-config-sample.php wp-cron.php wp-links-opml.php wp-login.php wp-settings.php wp-trackback.php
license.txt wordpress-5.2.5.tar.gz wp-admin wp-comments-post.php wp-content wp-includes wp-load.php wp-mail.php wp-signup.php xmlrpc.php
将该共享目录的权限设置nginx的启动账号nginx可读,Pod中nginx的UID为2020
root@ha1:/data/k8sdata/danran/wordpress# chown 2020.2020 ./* -R
root@ha1:/data/k8sdata/danran/wordpress# ll
在浏览器访问http://10.203.104.27:30031/wp-admin/setup-config.php测试
HA中设置wordpress的负载均衡地址
root@ha1:~# cat /etc/haproxy/haproxy.cfg
listen danran-nginx-80
bind 10.203.104.213:80
mode tcp
server master1 10.203.104.26:30031 check inter 3s fall 3 rise 5
server master2 10.203.104.27:30031 check inter 3s fall 3 rise 5
server master3 10.203.104.28:30031 check inter 3s fall 3 rise 5
root@ha1:~# systemctl restart haproxy.service
浏览器访问VIP地址测试(http://10.203.104.213/)
在本地hosts或DNS中添加VIP的主机名解析
10.203.104.213 www.linux.blog.com
测试主机访问wordpress(http://www.linux.blog.com/)
初始化WordPress站点
使用k8s中运行的mysql服务，作为mysql服务器，mysql服务器最好跟wordpress在同一NameSpace空间下
在mysql中创建wordpress数据库，并授权给wordpress用户
root@master1:~# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-0 2/2 Running 0 23h
mysql-1 2/2 Running 1 23h
mysql-2 2/2 Running 1 23h
root@master1:~# kubectl exec -it mysql-0 bash
Defaulting container name to mysql.
Use 'kubectl describe pod/mysql-0 -n default' to see all of the containers in this pod.
root@mysql-0:/# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 50586
Server version: 5.7.29-log MySQL Community Server (GPL)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CREATE DATABASE wordpress;
Query OK, 1 row affected (0.01 sec)
mysql> GRANT ALL PRIVILEGES ON wordpress.* TO "wordpress"@"%" IDENTIFIED BY "wordpress";
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> exit
Bye
k8s中测试MySQL连接
web界面初始化
mysql所在的NameSpace为Default,mysql的mysql-services.yaml文件中写了连接数据库需使用mysql-0.mysql，故mysql的全域名为mysql-0.mysql.default.svc.danran.com
Database Host 添加Mysql的主机地址(如果mysql pod不在同一个namespace，那么就写对方services全称，以实现跨namespace的解析和访问)
验证k8s中mysql 数据
分别验证k8s中MySQL主库和从库是否数据
danran
