kubernetes 1.17.2结合ceph13.2.8 实现jenkins部署并用traefik2.1代理
注：关于ceph、kubernetes集群的部署在此不声明，相信搜到本篇博文，你一定对ceph、kubernetes的部署环节手刃有余。
注：本篇博文牵扯到的技术点有：ceph、kubernetes、harbor、jenkins、traefik
ceph服务器操作
#ceph -s //查看ceph集群状态
#ceph osd pool create jenkins 128 //创建pool 建议每个pool存放的是通类应用
#ceph auth get-or-create client.jenkins mon 'allow r' osd 'allow class-read, allow rwx pool=jenkins' -o ceph.client.jenkins.keyring //创建普通用户管理对应pool
注意：ceph集群的状态要先调试成ok.
kubernetes拉取harbor镜像
# cat ~/.docker/config.json |base64 -w 0 //node节点访问私有仓库的认证
ewoJImF1dGhzIjogewoJCSJoYXJib3IubGludXguY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmVtbHpaV1psYVhwb2RRPT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjUgKGxpbnV4KSIKCX0KfQ==
# cat secret_harbor.yaml
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-02-19
#FileName: secret_harbor.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Secret
metadata:
name: k8s-harbor-login
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoYXJib3IubGludXguY29tIjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NmVtbHpaV1psYVhwb2RRPT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjUgKGxpbnV4KSIKCX0KfQ==
# kubectl create -f secret_harbor.yaml
secret/login created
# kubectl get secret
NAME TYPE DATA AGE
ceph-admin-secret kubernetes.io/rbd 1 3d16h
ceph-kube-secret kubernetes.io/rbd 1 3d16h
注意：前提是node节点可以访问到harbor，关于这部分可以参考我的这篇博文：https://www.cnblogs.com/zisefeizhu/p/12329864.html
部署动态存储
# pwd
/data/k8s/jenkins //单个服务单个目录
# cat namespace.yaml //单个服务单个名称空间，便于管理
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-09
#FileName: namespace.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Namespace
metadata:
name: jenkins
labels:
name: jenkins
#kubernetes结合ceph需要使用第三方插件
# cat external-storage-rbd-provisioner.yaml
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-09
#FileName: external-storage-rbd-provisioner.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
namespace: jenkins
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns"]
verbs: ["list", "get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: jenkins
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
namespace: jenkins
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
namespace: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: jenkins
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
namespace: jenkins
spec:
replicas: 1
selector:
matchLabels:
app: rbd-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
image: "harbor.linux.com/rbd/rbd-provisioner:latest"
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
imagePullSecrets:
- name: k8s-harbor-login
serviceAccount: rbd-provisioner
#敏感数据创建secret，这没什么可说的
# cat ceph-jenkins-secret.yaml
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-09
#FileName: ceph-wordpress-secret.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: v1
kind: Secret
metadata:
name: ceph-admin-secret
namespace: jenkins
data:
key: QVFBZ2pXVmVGOVJISkJBQTBTUDRoOTVZYVdHNEN6TzNaUWtIdVE9PQ==
type: kubernetes.io/rbd
---
apiVersion: v1
kind: Secret
metadata:
name: ceph-jenkins-secret
namespace: jenkins
data:
key: QVFEUjRHWmVNUFJpRnhBQUQ1Zlg1UG9JRUNkMG85Qk5kVzN5SUE9PQ==
type: kubernetes.io/rbd
# cat ceph-jenkins-storageclass.yaml //
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-09
#FileName: ceph-wordpress-storageclass.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-jenkins
namespace: jenkins
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: ceph.com/rbd
reclaimPolicy: Retain
parameters:
monitors: 20.0.0.207:6789,20.0.0.208:6789,20.0.0.210:6789
adminId: admin
adminSecretName: ceph-admin-secret
adminSecretNamespace: jenkins
pool: jenkins
fsType: xfs
userId: jenkins
userSecretName: ceph-jenkins-secret
imageFormat: "2"
imageFeatures: "layering"
# cat jenkins-pvc.yaml
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-10
#FileName: jenkins-pvc.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
namespace: jenkins
# labels:
# app: gitlab
spec:
storageClassName: ceph-jenkins
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
# kubectl get pvc -n jenkins
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-pvc Bound pvc-d386c125-5302-468a-8a94-a2570f0a4ca0 20Gi RWO ceph-jenkins 2d8h
部署jenkins应用
# pwd
/data/k8s/jenkins
# cat jenkins.yaml //核心资源清单
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: jenkins
namespace: jenkins
labels:
name: jenkins
spec:
selector:
matchLabels:
name: jenkins
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
#登陆私有仓库harbor认证
imagePullSecrets:
- name: k8s-harbor-login
containers:
- name: jenkins
image: harbor.linux.com/dev/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
- containerPort: 50000
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
# value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
securityContext:
fsGroup: 1000
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-pvc
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: jenkins
spec:
# type: LoadBalancer
selector:
name: jenkins
# ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
#externalTrafficPolicy: Local
ports:
-
name: http
port: 80
targetPort: 8080
protocol: TCP
-
name: agent
port: 50000
protocol: TCP
# cat jenkins-serviceaccount.yaml
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-10
#FileName: jenkins-serviceaccount.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
# In GKE need to get RBAC permissions first with
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
namespace: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
namespace: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: jenkins
# kubectl get pods -n jenkins
NAME READY STATUS RESTARTS AGE
jenkins-0 1/1 Running 14 30h
rbd-provisioner-5c97b9d5ff-95qwj 1/1 Running 13 2d8h
部署代理
# cat jenkins-ingressroute.yaml
##########################################################################
#Author: zisefeizhu
#QQ: 2********0
#Date: 2020-03-10
#FileName: jenkins-ingressroute.yaml
#URL: https://www.cnblogs.com/zisefeizhu/
#Description: The test script
#Copyright (C): 2020 All rights reserved
###########################################################################
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: jenkins
namespace: jenkins
spec:
entryPoints:
- web
routes:
#登陆域名 需要自己在主机添加hosts解析 或者自建dns也行
- match: Host(`jenkins.linux.com`)
kind: Rule
services:
- name: jenkins #和jenkins核心资源清单一致， name是jenkins service的name
port: 80 #jenkins pod 暴漏端口
登陆密码
# kubectl get pods -n jenkins
NAME READY STATUS RESTARTS AGE
jenkins-0 1/1 Running 1 18m
rbd-provisioner-dbc4c8b59-grfg2 1/1 Running 1 125m
# kubectl logs jenkins-0 -n jenkins
VM settings:
Max. Heap Size: 1.00G
Ergonomics Machine Class: server
Using VM: OpenJDK 64-Bit Server VM
#注意 jenkins首次登陆密码
ac4fe3940ec145fe9104eda3ca390d0a
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
*************************************************************
*************************************************************
2020-03-10 02:44:33.446+0000 [id=39] INFO hudson.model.UpdateSite#updateData: Obtained the latest update center data file for UpdateSource default
2020-03-10 02:44:36.699+0000 [id=25] INFO hudson.model.UpdateSite#updateData: Obtained the latest update center data file for UpdateSource default
2020-03-10 02:44:42.206+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-03-10 02:44:43.044+0000 [id=39] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
2020-03-10 02:44:43.044+0000 [id=39] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1
2020-03-10 02:44:43.055+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 81,778 ms
2020-03-10 02:44:43.414+0000 [id=19] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running
登陆密码为：
ac4fe3940ec145fe9104eda3ca390d0a
注：本篇博文完全原创，后续将发布大量有技术的原创博文，请持续关注。
注：关于ceph集群的部署、kubernetes1.17.2高可用集群的部署可以看我的历史博文。
注：关于jenkins的使用不在本篇讲解范围内(主要是截图太多了，有空再发表吧)
过手如登山，一步一重天
