使用Jenkins Operator在k8s中部署Jenkins
目录
1、Jenkins Operator介绍
Kubernetes Operator是一种特定于应用的控制器，可扩展Kubernetes API的功能，来代表Kubernetes用户创建、配置和管理复杂应用的实例
这里对Operator的相关介绍就不多赘述了，主要还是回到Jenkins Operator这个话题
基于k8s上Jenkins的常规安装是使用yaml资源清单，更为方便一点的是helm chart，但是常常我们在安装后还需要做很多的动作。例如插件问题，这尽管可以通过Configuration as Code的方式来解决，根据个人实际经验来看，还是存在一定几率会因主镜像版本、环境等存在诸多不可预知的问题。当然不同的部署方式都各有利弊，大家根据实际情况选择即可。
直到官方支持Jenkins可以在k8s中通过Operator方式部署，在4月中旬，Jenkins blog说道：Jenkins Operator 正式成为了 Jenkins 的子项目，填补了Jenkins与Kubernetes间的缝隙。也就是说，最初由（个人）三方团队编写的Jenkins Operator被Jenkins官方认可了
参考官方说明，Jenkins Operator可以帮我们解决以下问题：
-
安装指定版本的插件
即使最新版本插件不兼容或具备安全漏洞，还是为了插件稳定性而使用（因为常常会出现我们通过一键升级插件导致很多问题而去手动安装旧版本插件的情况）
-
更好的自定义配置
包含在安装指定版本插件时指定插件配置等声明式配置
-
开箱即用的安全配置
-
可灵活调整的
debug错误调试 -
备份和还原作业历史记录
......
2、Jenkins Operator的架构和设计
参考Jenkins Operator Architecture and design
Jenkins Operator的设计包含以下概念
- 监视清单的任何更改，并根据已部署的自定义资源清单维护所需的状态
- 实现主
reconciliation循环，由两个较小的reconciliation循环：base和user
Base reconciliation 循环负责监听Jenkins基础配置：
- 确认清单-监听清单中发生的任何更改
- 确保
Jenkins Pod状态，创建和验证Jenkins Server Pod的状态 - 确认
Jenkins的配置，包括安全加固、初始化配置等 - 确认
Jenkins API token，生成token并初始化Jenkins Client
User reconciliation循环负责协调用户提供的配置：
- 确保恢复任务，创建恢复任务，并确保恢复已成功执行
- 确保
Seed Jobs，创建Seed Jobs并确保所有这些工作都已成功执行 - 确保用户配置，执行用户提供的配置，如
groovy脚本，配置为代码或插件 - 确保备份任务，创建备份任务并确保备份成功
Operator状态
Operator状态保存在自定义资源状态部分中，该部分用于存储Operator管理的任何配置事件或Job状态
即使操作者或Jenkins重新启动，它也能帮助保持或恢复所需的状态
3、使用Operator部署Jenkins
3.1 前提条件
参考Jenkins Operator官方文档，需要有一个1.11+版本的Kubernetes集群，这里我的环境如下
# kubectl version -o yaml
clientVersion:
buildDate: "2020-12-08T17:59:43Z"
compiler: gc
gitCommit: af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38
gitTreeState: clean
gitVersion: v1.20.0
goVersion: go1.15.5
major: "1"
minor: "20"
platform: darwin/amd64
serverVersion:
buildDate: "2021-01-13T13:20:00Z"
compiler: gc
gitCommit: faecb196815e248d3ecfb03c680a4507229c2a56
gitTreeState: clean
gitVersion: v1.20.2
goVersion: go1.15.5
major: "1"
minor: "20"
platform: linux/amd64
3.2 获取并创建CRD
获取yaml并创建crd，当然也可以通过直接apply远程地址，这里先将其保存到本地
# wget -c https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/crds/jenkins_v1alpha2_jenkins_crd.yaml
# kubectl apply -f jenkins_v1alpha2_jenkins_crd.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/jenkins.jenkins.io created
customresourcedefinition.apiextensions.k8s.io/jenkinsimages.jenkins.io created
3.3 部署Jenkins Operator
有以下两种方式部署Jenkins Operator
-
使用
yaml一键安装，默认将安装在default命名空间下# kubectl apply -f https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/deploy/all-in-one-v1alpha2.yaml # kubectl get pods -w -
使用
helm并自定义安装，依赖helm在v3以上版本
创建ns
# kubectl create ns jenkins
添加helm仓库并获取chart
# helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart
"jenkins" has been added to your repositories
# helm pull jenkins/jenkins-operator
# tar xf jenkins-operator-0.4.3.tgz && cd jenkins-operator
修改value yaml部分内容，可以定义关于jenkins实例、operator deployment、backup备份相关、Configuration配置相关字段
- 指定ns
- 指定插件
- 默认情况只持久化了备份卷，这里将数据卷也做持久化，
sc使用csi-rbd-sc - 默认开启
configurationAsCode，并通过configmap和secret注入
jenkins:
...
namespace: jenkins
...
basePlugins:
- name: kubernetes
version: "1.28.6"
- name: workflow-job
version: "2.40"
- name: workflow-aggregator
version: "2.6"
- name: git
version: "4.5.0"
- name: job-dsl
version: "1.77"
- name: configuration-as-code
version: "1.46"
- name: kubernetes-credentials-provider
version: "0.15"
plugins:
- name: simple-theme-plugin
version: "0.6"
# plugins: []
...
# volumes used by Jenkins
# By default, we are only using backup
volumes:
- name: backup # PVC volume where backups will be stored
persistentVolumeClaim:
claimName: jenkins-backup
# volumeMounts are mounts for Jenkins pod
volumeMounts: []
...
backup:
# enabled is enable/disable switch for backup feature
# By default the feature is enabled
enabled: true
# image used by backup feature
# By default using prebuilt backup PVC image by VirtusLab
image: virtuslab/jenkins-operator-backup-pvc:v0.1.0
# containerName is backup container name
containerName: backup
# interval defines how often make backup in seconds
interval: 30
# makeBackupBeforePodDeletion when enabled will make backup before pod deletion
makeBackupBeforePodDeletion: true
# backupCommand is backup container command
backupCommand:
- /home/user/bin/backup.sh
# restoreCommand is backup restore command
restoreCommand:
- /home/user/bin/restore.sh
getLatestAction:
- /home/user/bin/get-latest.sh
# pvc is Persistent Volume Claim Kubernetes resource
pvc:
# enabled is enable/disable switch for PVC
enabled: true
# size is size of PVC
size: 5Gi
# className is storageClassName for PVC
# See https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1 for more details
className: "csi-rbd-sc"
# env contains container environment variables
# PVC backup provider handles these variables:
# BACKUP_DIR - path for storing backup files (default: "/backup")
# JENKINS_HOME - path to jenkins home (default: "/jenkins-home")
# BACKUP_COUNT - define how much recent backups will be kept
env:
- name: BACKUP_DIR
value: /backup
- name: JENKINS_HOME
value: /jenkins-home
- name: BACKUP_COUNT
value: "3" # keep only the 3 most recent backups
# volumeMounts holds the mount points for volumes
volumeMounts:
- name: jenkins-home
mountPath: /jenkins-home # Jenkins home volume
- mountPath: /backup # backup volume
name: backup
...
configuration:
configurationAsCode: {}
# - configMapName: jenkins-casc
# content: {}
groovyScripts: {}
# - configMapName: jenkins-gs
# content: {}
# secretRefName of existing secret (previously created)
secretRefName: ""
# secretData creates new secret if secretRefName is empty and fills with data provided in secretData
secretData: {}
执行安装
# helm install jenkins jenkins-operator -n jenkins --values ./jenkins-operator/values.yaml
NAME: jenkins
LAST DEPLOYED: Sun May 16 19:42:32 2021
NAMESPACE: jenkins
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch Jenkins instance being created:
$ kubectl --namespace jenkins get pods -w
2. Get Jenkins credentials:
$ kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.user}' | base64 -d
$ kubectl --namespace jenkins get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.password}' | base64 -d
3. Connect to Jenkins (actual Kubernetes cluster):
$ kubectl --namespace jenkins port-forward jenkins-jenkins 8080:8080
Now open the browser and enter http://localhost:8080
检查创建的operator
# kubectl get pods -n jenkins
NAME READY STATUS RESTARTS AGE
jenkins-jenkins 1/2 Running 0 44s
jenkins-jenkins-operator-996887c4b-wftz2 1/1 Running 0 1m29s
# kubectl -n jenkins get jenkins
NAME AGE
jenkins 70s
3.4 部署Jenkins
一旦上面的Jenkins Operator部署后启动并正常运行，就自动会部署一个Jenkins实例Pod了
实际上可以看到，通过Jenkins Operator部署的Jenkins的控制器不是场景k8s自带的三大控制器，而是由operator自己管控
观察operator的日志如下
# kubectl -n jenkins logs -f jenkins-jenkins-operator-996887c4b-wftz2
2021-05-16T11:59:05.017Z INFO controller-jenkins jenkins/jenkins_controller.go:432 Setting default Jenkins API settings {"cr": "jenkins"}
2021-05-16T11:59:05.073Z INFO controller-jenkins jenkins/handler.go:88 *v1alpha2.Jenkins/jenkins has been updated {"cr": "jenkins"}
2021-05-16T11:59:06.568Z INFO controller-jenkins base/pod.go:159 Creating a new Jenkins Master Pod jenkins/jenkins-jenkins {"cr": "jenkins"}
观察jenkins pod中jenkins master的日志如下，正在下载插件（此步骤稍慢）
# kubectl -n jenkins logs -f jenkins-jenkins -c jenkins-master
...
> bootstrap4-api depends on font-awesome-api:5.15.2-2,jquery3-api:3.5.1-3,popper-api:1.16.1-2
Downloading plugin: font-awesome-api from https://updates.jenkins.io/dynamic-2.263//latest/font-awesome-api.hpi
Downloading plugin: jquery3-api from https://updates.jenkins.io/dynamic-2.263//latest/jquery3-api.hpi
Downloading plugin: popper-api from https://updates.jenkins.io/dynamic-2.263//latest/popper-api.hpi
如果在有限时间（健康检查时间）内没有下载成功，这通常是由于网络原因引起的，Operator会中断该Pod并重新创建
# kubectl -n jenkins logs -f jenkins-jenkins-operator-996887c4b-wftz2
2021-05-16T12:09:42.854Z INFO controller-jenkins base/reconcile.go:370 Container 'jenkins-master' is terminated, status '{Name:jenkins-master State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:137,Signal:0,Reason:Error,Message:,StartedAt:2021-05-16 12:05:54 +0000 UTC,FinishedAt:2021-05-16 12:09:42 +0000 UTC,ContainerID:docker://342349d9e4045cd312345937797a2d9f048f3623fc1ce6bf1c2b77ff4f04d8da,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:jenkins/jenkins:2.263.2-lts-alpine ImageID:docker-pullable://jenkins/jenkins@sha256:496142509b7d3e3f22f5cdc81b1d1322db61ec929d34dfd66b9ec3257bca13e5 ContainerID:docker://342349d9e4045cd312345937797a2d9f048f3623fc1ce6bf1c2b77ff4f04d8da Started:0xc0005aa2c6}' {"cr": "jenkins"}
可行的一个解决办法是将value.yaml中的健康检查时间微调或者临时去掉健康检查，并helm更新让其正常启动并持久化后再次恢复，或者新创建一个Jenkins控制器将其覆盖
# helm -n jenkins upgrade jenkins jenkins-operator --values ./jenkins-operator/values.yaml
最终直到看见这样的日志，就表示Jenkins启动成功了
2021-05-16 13:26:14.221+0000 [id=28] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@295b7e33]: org.springframework.beans.factory.support.DefaultListableBeanFactory@52880f75
2021-05-16 13:26:14.223+0000 [id=28] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@52880f75: defining beans [filter,legacy]; root of factory hierarchy
2021-05-16 13:26:14.489+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-05-16 13:26:14.767+0000 [id=20] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
到这里，通过Jenkins Operator部署Jenkins就完成了（尽管看上去也没多少比helm或传统方式部署的优势），其实Jenkins Operator还有更为好用的的其他功能，后续再介绍。
See you ~
