linux运维、架构之路-K8s中部署Jenkins集群高可用
一、在K8s中 部署Jenkins优点
1、传统Jenkins集群架构一些问题
- Master发生故障时，整个流程都不可用
- Slave集群的环境配置不一样，来完成不同语言的编译打包，但是这些差异化的配置导致管理起来不方便，维护麻烦
- 资源分配不均衡，有的slave要运行的job出现排队等待，而有的salve处于空闲状态
- 资源浪费，每台slave可能是物理机或者虚拟机，当slave处于空闲状态时，也不能完全释放掉资源
2、K8s中Jenkins集群架构优点
- 当Jenkins Master接受到Build请求后，会根据配置的Label动态创建一个运行在Pod中的Jenkins Slave并注册到Master上，当运行完Job后，这个Slave会被注销并且这个Pod也会自动删除，恢复到最初的状态(这个策略可以设置)
- 服务高可用，当Jenkins Master出现故障时，Kubernetes会自动创建一个新的Jenkins Master容器，并且将Volume分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用的作用
- 动态伸缩，合理使用资源，每次运行Job时，会自动创建一个Jenkins Slave，Job完成后，Slave自动注销并删除容器，资源自动释放，并且Kubernetes会根据每个资源的使用情况，动态分配slave到空闲的节点上创建，降低出现因某节点资源利用率高，降低出现因某节点利用率高出现排队的情况
- 扩展性好，当Kubernetes集群的资源严重不足导致Job排队等待时，可以很容器的添加一个Kubernetes Node到集群，从而实现扩展
二、K8s 部署Jenkins
1、集群环境
2、k8s-node1部署NFS服务
①NFS服务端配置
mkdir -p /data/k8s chown -R nfsnobody.nfsnobody /data echo "/data/k8s 192.168.56.0/24(rw,no_root_squash,sync)" >/etc/exports systemctl enable rpcbind systemctl enable nfs systemctl start rpcbind systemctl start nf
②客户端挂载
systemctl start rpcbind systemctl enable rpcbind mkdir /data/k8s -p mount -t nfs 192.168.56.11:/data/k8s /data/k8
3、创建Jenkins集群所需的YAML文件
①创建命名空间和存放Jenkins的YAML目录
kubectl create namespace demon mkdir -p /opt/jenkins
②为Jenkins数据持久化存储创建一个PV
cat >/opt/jenkins/jenkins_pv.yaml <<EOF apiVersion: v1 kind: PersistentVolume metadata: name: opspv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete nfs: server: 192.168.56.11 path: /data/k8s --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: opspvc namespace: demon spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi EOF
③创建Jenkins集群权限serviceAccount文件
cat >/opt/jenkins/jenkins_rbac.yaml <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: demon --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: jenkins namespace: demon roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: demon EOF
④创建Jenkins Deployment
cat jenkins_deployment.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: jenkins namespace: demon spec: template: metadata: labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccount: jenkins containers: - name: jenkins image: jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080 #外部访问端口 name: web protocol: TCP - containerPort: 50000 #jenkins save发现端口 name: agent protocol: TCP resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 500m memory: 512Mi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 #容器初始化完成后，等待60秒进行探针检查 timeoutSeconds: 5 failureThreshold: 12 #当Pod成功启动且检查失败时，Kubernetes将在放弃之前尝试failureThreshold次。放弃生存检查意味着重新启动Pod。而放弃就绪检查，Pod将被标记为未就绪。默认为3.最小值为1 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 volumeMounts: #需要将jenkins_home目录挂载出来 - name: jenkinshome subPath: jenkins mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: opspvc
⑤创建Jenkins SVC
cat >/opt/jenkins/jenkins_svc.yaml <<EOF apiVersion: v1 kind: Service metadata: name: jenkins namespace: demon labels: app: jenkins spec: selector: app: jenkins type: NodePort ports: - name: web port: 8080 targetPort: web nodePort: 30002 - name: agent port: 50000 targetPort: agent EOF
⑥依次创建
[root@k8s-node1 jenkins]# ll 总用量 16 -rw-r--r-- 1 root root 2281 12月 27 14:59 jenkins_deployment.yaml -rw-r--r-- 1 root root 404 12月 27 14:57 jenkins_pv.yaml -rw-r--r-- 1 root root 1108 12月 27 14:58 jenkins_rbac.yaml -rw-r--r-- 1 root root 285 12月 27 14:59 jenkins_svc.yaml
[root@k8s-node1 jenkins]# kubectl apply -f jenkins_pv.yaml persistentvolume/opspv created persistentvolumeclaim/opspvc created [root@k8s-node1 jenkins]# kubectl apply -f jenkins_rbac.yaml serviceaccount/jenkins created clusterrole.rbac.authorization.k8s.io/jenkins created clusterrolebinding.rbac.authorization.k8s.io/jenkins created [root@k8s-node1 jenkins]# kubectl apply -f jenkins_deployment.yaml deployment.extensions/jenkins created [root@k8s-node1 jenkins]# kubectl apply -f jenkins_svc.yaml service/jenkins create
⑦查看结果
[root@k8s-node1 jenkins]# kubectl get pv,pvc,pod,svc -n demon NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/opspv 10Gi RWX Delete Bound demon/opspvc 1h NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/opspvc Bound opspv 10Gi RWX 1h NAME READY STATUS RESTARTS AGE pod/jenkins-6d7bc49b74-d9jxc 1/1 Running 0 1h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/jenkins NodePort 10.1.148.201 <none> 8080:30002/TCP,50000:26723/TCP 1h
8080：端口为我们jenkins访问端口 50000：端口为jenkins save发现端口
⑧通过浏览器访问集群任意IP的svc端口
管理员密码路径：持久化在/data/k8s下，所以jenkins的所有配置都在这下面
cat /data/k8s/jenkins/secrets/initialAdminPassword
直接推荐安装即可
⑨安装完成后我们进入jenkins主页面
⑩enkins–>插件–>安装插件Kubernetes
三、Jenkins中配置k8s
1、系统管理->系统配置
2、配置拉到最下面找到Kubernetes插件
Name 配置的名称
Kubernetes URL 这里的URL是K8s内部的URL，实际上就是svcname https://kubernetes.default.svc.cluster.local
Kubernetes Namespace k8s的命名空间 （实际上就是Jenkins所在的命名空间）
3、Jenkins URL配置
Jenkins URL 这里的URL是jenkins的svc名称加上命名空间，实际上就是在k8s集群内部访问jenkins的一个方法，这里也不需要修改
http://jenkins.demon.svc.cluster.local:8080
4、配置添 Jenkins Slave Pod模板
Name = Pod 名称 Namespave = Pod命名空间 Labels = Pod标签
5、容器的模板配置
6、创建volume的配置
Jenkins Master收到Build请求时，会根据配置的Label动态创建一个运行在Pod中的Jenkins Slave并注册到Master上，当Job运行完，这个Slave会被注销并且这个Pod也会自动删除，恢复到最初状态
四、测试验证
1、新建Job选择流水线
2、流水线Pipeline
def label = "jenkins-slave" podTemplate(label: label, cloud: 'kubernetes') { node(label) { stage('pull code') { echo "拉取代码" } stage('build') { echo "代码编译" } stage('SonarQube') { echo "质量扫描" } } }
3、执行效果
成功最有效的方法就是向有经验的人学习！
