基于Kubernetes/K8S构建Jenkins持续集成平台
基于Kubernetes/K8S构建Jenkins持续集成平台
作者：运维人在路上
个人博客：https://www.cnblogs.com/hujinzhong
微信公众号：运维人在路上
Bilibili账号：https://space.bilibili.com/409609392
个人简介：本人是一枚大型电商平台的运维工程师，对开发及运维有一定了解，现分享技术干货，欢迎大家交流！
一、Kubernetes介绍
1.1、简介
Kubernetes（简称，K8S）是Google开源的容器集群管理系统，在Docker技术的基础上，为容器化的应用提供部署运行、资源调度、服务发现和动态伸缩等一系列完整功能，提高了大规模容器集群管理的便捷性。 其主要功能如下：
- 使用Docker对应用程序包装(package)、实例化(instantiate)、运行(run)。
- 以集群的方式运行、管理跨机器的容器。
- 解决Docker跨机器容器之间的通讯问题
- Kubernetes的自我修复机制使得容器集群总是运行在用户期望的状态。
1.2、Kubernetes架构
master节点：
API Server：用于暴露Kubernetes API，任何资源的请求的调用操作都是通过kube-apiserver提供的接口进行的。Etcd：是Kubernetes提供默认的存储系统，保存所有集群数据，使用时需要为etcd数据提供备份计划。Controller-Manager：作为集群内部的管理控制中心，负责集群内的Node、Pod副本、服务端点（Endpoint）、命名空间（Namespace）、服务账号（ServiceAccount）、资源定额（ResourceQuota）的管理，当某个Node意外宕机时，Controller Manager会及时发现并执行自动化修复流程，确保集群始终处于预期的工作状态。Scheduler：监视新创建没有分配到Node的Pod，为Pod选择一个Node。
node节点：
Kubelet：负责维护容器的生命周期，同时负责Volume和网络的管理Kube proxy：是Kubernetes的核心组件，部署在每个Node节点上，它是实现Kubernetes Service的通信与负载均衡机制的重要组件
二、Kubernates集群安装
2.1、架构图
大致工作流程：手动/自动构建 -> Jenkins 调度 K8S API -＞动态生成 Jenkins Slave pod -＞ Slave pod拉取 Git 代码／编译／打包镜像 -＞推送到镜像仓库 Harbor -＞ Slave 工作完成，Pod 自动销毁 -＞部署到测试或生产 Kubernetes平台。（完全自动化，无需人工干预）
2.2、部署方案好处
1）服务高可用：当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用。
2）动态伸缩，合理使用资源：每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。
3）扩展性好：当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个Kubernetes Node 到集群中，从而实现扩展。
2.3、服务器规划
| 主机名称 | ip地址 | 安装的软件 |
|---|---|---|
| 代码托管服务器 | 10.0.0.101 | Gitlab |
| Docker镜像仓库服务器 | 10.0.0.102 | Harbor |
| k8s-master | 10.0.0.103 | kube-apiserver、kube-controller-manager、kube-scheduler、docker、etcd、calico，NFS |
| k8s-node1 | 10.0.0.104 | kubelet、kubeproxy、Docker |
| k8s-node2 | 10.0.0.105 | kubelet、kubeproxy、Docker |
2.4、k8s部署
2.4.1、前置环境
三台服务器都需要完成
1）修改三台机器的hostname及hosts文件
# 修改hostname
hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2
# 修改hosts文件
cat >>/etc/hosts <<EOF
10.0.0.103 k8s-master
10.0.0.104 k8s-node1
10.0.0.105 k8s-node2
EOF
2）关闭防火墙和selinux（云服务器无需配置）
# 防火墙
systemctl stop firewalld
systemctl disable firewalld
# selinux
setenforce 0 # 临时关闭
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux # 永久关闭
3）设置系统参数
# 设置允许路由转发，不对bridge的数据进行处理
cat >>/etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF
# 执行生效
sysctl -p /etc/sysctl.d/k8s.conf
4）kube-proxy开启ipvs的前置条件
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
5）所有节点关闭swap（云服务器无需配置）
# 临时关闭
[root@k8s-master ~]# swapoff -a
# 永久关闭（注释swap行）
[root@k8s-master ~]# vim /etc/fstab
#UUID=70e8d1ea-c8a2-4cc4-8146-fe2ef7229b4b swap swap defaults 0 0
6）所有节点安装docker
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#g' /etc/yum.repos.d/docker-ce.repo
yum install docker-ce lrzsz -y
docker version
mkdir -p /etc/docker/ && touch /etc/docker/daemon.json
cat >/etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["https://t09ww3n9.mirror.aliyuncs.com"],
"insecure-registries": ["10.0.0.101:85"]
}
EOF
systemctl start docker
systemctl enable docker
7）所有节点安装kubelet、kubeadm、kubectl
kubeadm: 用来初始化集群的指令。kubelet: 在集群中的每个节点上用来启动 pod 和 container 等。kubectl: 用来与集群通信的命令行工具。
# 清空yum缓存
yum clean all
# 设置yum安装源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 安装指定版本
yum install -y kubelet-1.17.0 kubeadm-1.17.0 kubectl-1.17.0
# kubelet设置开机启动（注意：先不启动，现在启动的话会报错）
systemctl enable kubelet
# 查看版本
kubelet --version
2.4.2、master节点初始化
# 1、运行初始化命令(注意：apiserver-advertise-address这个地址必须是master机器的IP)
kubeadm init --kubernetes-version=1.17.0 \
--apiserver-advertise-address=10.0.0.103 \
--image-repository registry.aliyuncs.com/google_containers \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16
# 该命令需要先记录下来，后续node节点加入需要使用
kubeadm join 10.0.0.103:6443 --token 8ihlea.z732b48fmwxtow6v \
--discovery-token-ca-cert-hash sha256:a54882ca516cb6615651a4cb5743d9e6733fa9335e46b1d030f6e3f16538e203
# 2、配置kubectl工具
[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 3、启动kubelet
[root@k8s-master k8s]# systemctl start kubelet
# 4、安装calico
[root@k8s-master k8s]# mkdir k8s
[root@k8s-master k8s]# cd k8s
[root@k8s-master k8s]# wget https://docs.projectcalico.org/v3.10/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
[root@k8s-master k8s]# sed -i 's/192.168.0.0/10.244.0.0/g' calico.yaml
[root@k8s-master k8s]# kubectl apply -f calico.yaml
# 5、等待几分钟，查看所有Pod的状态，确保所有Pod都是Running状态
[root@k8s-master k8s]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 10m v1.17.0
[root@k8s-master k8s]# kubectl get pod --all-namespaces -o wide
2.4.3、slave节点加入集群
# 1、让所有节点让集群环境，使用之前Master节点产生的命令加入集群
kubeadm join 10.0.0.103:6443 --token 8ihlea.z732b48fmwxtow6v \
--discovery-token-ca-cert-hash sha256:a54882ca516cb6615651a4cb5743d9e6733fa9335e46b1d030f6e3f16538e203
# 2、启动kubelet
[root@k8s-node1 ~]# systemctl start kubelet
[root@k8s-node1 ~]# systemctl status kubelet
# 3、回到Master节点查看，如果Status全部为Ready，代表集群环境搭建成功
[root@k8s-master k8s]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 23m v1.17.0
k8s-node1 Ready <none> 2m45s v1.17.0
k8s-node2 Ready <none> 67s v1.17.0
2.4.4、kubectl常用命令
kubectl get nodes # 查看所有主从节点的状态
kubectl get ns # 获取所有namespace资源
kubectl get pods -n {$nameSpace} # 获取指定namespace的pod
kubectl describe pod的名称 -n {$nameSpace} # 查看某个pod的执行过程
kubectl logs --tail=1000 pod的名称 | less # 查看日志
kubectl create -f xxx.yml # 通过配置文件创建一个集群资源对象
kubectl delete -f xxx.yml # 通过配置文件删除一个集群资源对象
kubectl delete pod名称 -n {$nameSpace} # 通过pod删除集群资源
kubectl get service -n {$nameSpace} # 查看pod的service情况
三、Kubernetes/Jenkins持续集成部署
3.1、NFS安装配置
3.1.1、NFS简介
NFS（Network File System），它最大的功能就是可以通过网络，让不同的机器、不同的操作系统可以共享彼此的文件。我们可以利用NFS共享Jenkins运行的配置文件、Maven的仓库依赖文件等
3.1.2、NFS安装
我们把NFS服务器安装在master节点上
# 1、安装NFS服务（在所有K8S的节点都需要安装）
[root@k8s-master ~]# yum install -y nfs-utils
# 2、创建共享目录
[root@k8s-master ~]# mkdir -p /opt/nfs/jenkins
[root@k8s-master ~]# vim /etc/exports
/opt/nfs/jenkins *(rw,no_root_squash) # *代表对所有IP都开放此目录，rw是读写
# 3、启动服务
[root@k8s-master ~]# systemctl enable nfs
[root@k8s-master ~]# systemctl start nfs
# 4、查看NFS共享目录
[root@k8s-node1 ~]# showmount -e 10.0.0.103
Export list for 10.0.0.103:
/opt/nfs/jenkins *
3.2、kubernetes安装Jenkins-Master
3.2.1、创建NFS client provisioner
nfs-client-provisioner 是一个Kubernetes的简易NFS的外部provisioner，本身不提供NFS，需要现有的NFS服务器提供存储
1）上传nfs-client-provisioner构建文件
[root@k8s-master nfs-client]# ll
total 12
-rw-r--r-- 1 root root 225 Nov 27 2019 class.yaml
-rw-r--r-- 1 root root 985 Dec 9 2019 deployment.yaml
-rw-r--r-- 1 root root 1526 Nov 27 2019 rbac.yaml
# 注意修改deployment.yaml，使用之前配置NFS服务器和目录
自定义存储资源清单：class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "true"
rbac权限资源清单：rbac.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
自定义deployment：deployment.yaml，注意使用之前配置NFS服务器和目录
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: lizhenliang/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.0.0.103
- name: NFS_PATH
value: /opt/nfs/jenkins/
volumes:
- name: nfs-client-root
nfs:
server: 10.0.0.103
path: /opt/nfs/jenkins/
2）构建nfs-client-provisioner的pod资源
[root@k8s-master nfs-client]# pwd
/root/k8s/nfs-client
[root@k8s-master nfs-client]# kubectl create -f .
[root@k8s-master nfs-client]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-989bf44d-gtn4k 1/1 Running 0 110m
3.2.2、安装Jenkins-Master
1）上传Jenkins-Master构建文件
[root@k8s-master jenkins-master]# pwd
/root/k8s/jenkins-master
[root@k8s-master jenkins-master]# ll
total 16
-rw-r--r-- 1 root root 1874 Dec 8 2019 rbac.yaml
-rw-r--r-- 1 root root 87 Dec 7 2019 ServiceAcount.yaml
-rw-r--r-- 1 root root 284 Dec 7 2019 Service.yaml
-rw-r--r-- 1 root root 2116 Dec 7 2019 StatefulSet.yaml
自定义rbac权限管理：rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
namespace: kube-ops
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
namespace: kube-ops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkinsClusterRole
namespace: kube-ops
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkinsClusterRuleBinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkinsClusterRole
subjects:
- kind: ServiceAccount
name: jenkins
namespace: kube-ops
自定义ServiceaAcount：ServiceaAcount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: kube-ops
自定义Service：Service.yaml
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: kube-ops
labels:
app: jenkins
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
自定义有状态应用：StatefulSet.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
namespace: kube-ops
spec:
serviceName: jenkins
selector:
matchLabels:
app: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
app: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
image: jenkins/jenkins:lts-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
securityContext:
fsGroup: 1000
volumeClaimTemplates:
- metadata:
name: jenkins-home
spec:
storageClassName: "managed-nfs-storage"
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
注意以下两点：
- 在
StatefulSet.yaml文件，声明了利用nfs-client-provisioner进行Jenkins-Master文件存储 - Service发布方法采用
NodePort，会随机产生节点访问端口
2）创建kube-ops的namespace
# 把Jenkins-Master的pod放到kube-ops下
[root@k8s-master jenkins-master]# kubectl create namespace kube-ops
namespace/kube-ops create
3）构建Jenkins-Master的pod资源
[root@k8s-master jenkins-master]# pwd
/root/k8s/jenkins-master
[root@k8s-master jenkins-master]# kubectl create -f .
[root@k8s-master jenkins-master]# kubectl get pods -n kube-ops
NAME READY STATUS RESTARTS AGE
jenkins-0 0/1 Running 0 77s
4）查看pod的相关信息
# 查看Pod运行在那个Node上
[root@k8s-master jenkins-master]# kubectl describe pods -n kube-ops
# 查看分配的端口
[root@k8s-master jenkins-master]# kubectl get service -n kube-ops
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins NodePort 10.1.228.200 <none> 8080:30707/TCP,50000:32495/TCP 2m45s
5）浏览器访问
访问地址：http://10.0.0.105:30707/，ip为node2的ip地址
[root@k8s-master secrets]# pwd
/opt/nfs/jenkins/kube-ops-jenkins-home-jenkins-0-pvc-c64f6c53-01a3-4b5d-bc76-e61f3cb0116a/secrets
[root@k8s-master secrets]# cat initialAdminPassword
1b0ea53845fa48ef9d5a8e5017a052ca
安装过程与之前一样
6）修改jenkins插件下载地址
[root@k8s-master updates]# pwd
/opt/nfs/jenkins/kube-ops-jenkins-home-jenkins-0-pvc-c64f6c53-01a3-4b5d-bc76-e61f3cb0116a/updates
[root@k8s-master updates]# sed -i 's/https:\/\/updates.jenkins.io\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json
[root@k8s-master updates]# sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json
最后，Manage Plugins点击Advanced，把Update Site改为国内插件下载地址后重启Jenkins:
https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
7）安装基本插件
# 基本插件如下：
Localization:Chinese
Git
Pipeline
Extended Choice Parameter
3.3、Jenkins与Kubernetes整合
1）安装Kubernetes插件
2）实现jenkins与kubernetes整合
系统管理->系统配置->云->新建云->Kubernetes
说明：
kubernetes地址采用了kube的服务器发现：https://kubernetes.default.svc.cluster.local
namespace填
kube-ops，然后点击Test Connection，如果出现成功的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信Jenkins URL 地址：http://jenkins.kube-ops.svc.cluster.local:8080
3.4、构建Jenkins-Slave自定义镜像
Jenkins-Master在构建Job的时候，Kubernetes会创建Jenkins-Slave的Pod来完成Job的构建。我们选择运行Jenkins-Slave的镜像为官方推荐镜像：jenkins/jnlp-slave:latest，但是这个镜像里面并没有Maven环境，为了方便使用，我们需要自定义一个新的镜像：
[root@k8s-master jenkins-slave]# ll
total 8948
-rw-r--r-- 1 root root 9142315 Nov 8 2019 apache-maven-3.6.2-bin.tar.gz
-rw-r--r-- 1 root root 556 Dec 20 2019 Dockerfile
-rw-r--r-- 1 root root 10475 Nov 25 2019 settings.xml # 添加阿里云maven仓库
Dockerfifile文件内容如下：
FROM jenkins/jnlp-slave:latest
MAINTAINER itcast
# 切换到 root 账户进行操作
USER root
# 安装 maven
COPY apache-maven-3.6.2-bin.tar.gz .
RUN tar -zxf apache-maven-3.6.2-bin.tar.gz && \
mv apache-maven-3.6.2 /usr/local && \
rm -f apache-maven-3.6.2-bin.tar.gz && \
ln -s /usr/local/apache-maven-3.6.2/bin/mvn /usr/bin/mvn && \
ln -s /usr/local/apache-maven-3.6.2 /usr/local/apache-maven && \
mkdir -p /usr/local/apache-maven/repo
COPY settings.xml /usr/local/apache-maven/conf/settings.xml
USER jenkins
构建出一个新镜像：jenkins-slave-maven:latest，把镜像上传到Harbor的公共库library中
# 构建镜像
[root@k8s-master jenkins-slave]# docker build -t jenkins-slave-maven:latest .
[root@k8s-master jenkins-slave]# docker images|grep "jenkins-slave-maven"
# 登录Harbor
[root@k8s-master jenkins-slave]# docker login -u admin -p Harbor12345 10.0.0.101:85
# 打标签并推送镜像（放在library公共项目中）
[root@k8s-master jenkins-slave]# docker tag jenkins-slave-maven:latest 10.0.0.101:85/library/jenkins-slave-maven:latest
[root@k8s-master jenkins-slave]# docker push 10.0.0.101:85/library/jenkins-slave-maven:latest
3.5、测试Jenkins-Slave是否可以创建
1）jenkins中创建gitlab凭证
2）创建一个流水线项目
3）编写pipeline，从gitlab拉取代码
def git_address = "http://10.0.0.101:82/dianchou_group/tensqure_back.git"
def git_auth = "af309a92-668d-4ba9-bb14-af09d398f24b"
//创建一个Pod的模板，label为jenkins-slave
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: "10.0.0.101:85/library/jenkins-slave-maven:latest"
)
]
)
{
//引用jenkins-slave的pod模块来构建Jenkins-Slave的pod
node("jenkins-slave"){
// 第一步
stage('拉取代码'){
checkout([$class: 'GitSCM', branches: [[name: 'master']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
}
}
4）控制台输入
3.6、微服务持续集成
3.6.1、拉取代码构建镜像
1）创建NFS共享目录，让所有Jenkins-Slave构建指向NFS的Maven的共享仓库目录
[root@k8s-master ~]# vim /etc/exports
/opt/nfs/jenkins *(rw,no_root_squash)
/opt/nfs/maven *(rw,no_root_squash)
[root@k8s-master ~]# systemctl restart nfs
2）创建harbor凭证
3）创建流水线项目并编写pipeline
添加字符参数：
添加多选框参数：
pipeline脚本：
def git_address = "http://10.0.0.101:82/dianchou_group/tensqure_back.git"
def git_auth = "af309a92-668d-4ba9-bb14-af09d398f24b"
//构建版本的名称
def tag = "latest"
//Harbor私服地址
def harbor_url = "10.0.0.101:85"
//Harbor的项目名称
def harbor_project_name = "dianchou"
//Harbor的凭证
def harbor_auth = "23a9742f-e72d-4541-aa63-6389c3a30828"
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: "10.0.0.101:85/library/jenkins-slave-maven:latest"
),
containerTemplate(
name: 'docker',
image: "docker:stable",
ttyEnabled: true,
command: 'cat'
),
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '10.0.0.103' , serverPath: '/opt/nfs/maven'),
],
)
{
node("jenkins-slave"){
// 第一步
stage('拉取代码'){
checkout([$class: 'GitSCM', branches: [[name: '${branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
// 第二步
stage('代码编译'){
//编译并安装公共工程
sh "mvn -f tensquare_common clean install"
}
// 第三步
stage('构建镜像，部署项目'){
//把选择的项目信息转为数组
def selectedProjects = "${project_name}".split(',')
for(int i=0;i<selectedProjects.size();i++){
//取出每个项目的名称和端口
def currentProject = selectedProjects[i];
//项目名称
def currentProjectName = currentProject.split('@')[0]
//项目启动端口
def currentProjectPort = currentProject.split('@')[1]
//定义镜像名称
def imageName = "${currentProjectName}:${tag}"
//编译，构建本地镜像
sh "mvn -f ${currentProjectName} clean package dockerfile:build"
container('docker') {
//给镜像打标签
sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${imageName}"
//登录Harbor，并上传镜像
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
//登录
sh "docker login -u ${username} -p ${password} ${harbor_url}"
//上传镜像
sh "docker push ${harbor_url}/${harbor_project_name}/${imageName}"
}
//删除本地镜像
sh "docker rmi -f ${imageName}"
sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${imageName}"
}
}
}
}
}
需要注意的坑：
a、在构建过程会发现无法创建仓库目录，是因为NFS共享目录权限不足，需更改权限
chown -R jenkins:jenkins /opt/nfs/maven
chmod -R 777 /opt/nfs/maven
b、报错：java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.io.IOException: Permission denied
# Docker命令执行权限问题
chmod 777 /var/run/docker.sock
c、需要手动上传父工程依赖到NFS的Maven共享仓库目录中
[root@k8s-master tensquare]# pwd
/opt/nfs/maven/com/tensquare
[root@k8s-master tensquare]# ll
total 0
drwxrwxrwx 3 jenkins jenkins 58 Mar 25 17:50 tensquare_common
drwxrwxrwx 3 jenkins jenkins 58 Mar 25 18:05 tensquare_parent # 手动添加
3.6.2、微服务部署到k8s
1）修改每个微服务的application.yml
以eureka为例
server:
port: ${PORT:10086}
spring:
application:
name: eureka
eureka:
server:
# 续期时间，即扫描失效服务的间隔时间（缺省为60*1000ms）
eviction-interval-timer-in-ms: 5000
enable-self-preservation: false
use-read-only-response-cache: false
client:
# eureka client间隔多久去拉取服务注册信息 默认30s
registry-fetch-interval-seconds: 5
serviceUrl:
defaultZone: ${EUREKA_SERVER:http://127.0.0.1:${server.port}/eureka/}
instance:
# 心跳间隔时间，即发送一次心跳之后，多久在发起下一次（缺省为30s）
lease-renewal-interval-in-seconds: 5
# 在收到一次心跳之后，等待下一次心跳的空档时间，大于心跳间隔即可，即服务续约到期时间（缺省为90s）
lease-expiration-duration-in-seconds: 10
instance-id: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}:${server.port}@${random.long(1000000,9999999)}
hostname: ${EUREKA_INSTANCE_HOSTNAME:${spring.application.name}}
其他服务需要注册到所有Eureka中：
# Eureka配置
eureka:
client:
service-url:
defaultZone: http://eureka-0.eureka:10086/eureka/,http://eureka-1.eureka:10086/eureka/ # Eureka访问地址
instance:
prefer-ip-address: true
2）安装Kubernetes Continuous Deploy插件
3）修改流水线脚本，添加部署步骤
def git_address = "http://10.0.0.101:82/dianchou_group/tensqure_back.git"
def git_auth = "af309a92-668d-4ba9-bb14-af09d398f24b"
//构建版本的名称
def tag = "latest"
//Harbor私服地址
def harbor_url = "10.0.0.101:85"
//Harbor的项目名称
def harbor_project_name = "dianchou"
//Harbor的凭证
def harbor_auth = "23a9742f-e72d-4541-aa63-6389c3a30828"
def secret_name = "registry-auth-secret"
//k8s凭证
def k8s_auth = "fb395f34-03a6-4e40-b0b4-349d8cd45a34";
podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
containerTemplate(
name: 'jnlp',
image: "10.0.0.101:85/library/jenkins-slave-maven:latest"
),
containerTemplate(
name: 'docker',
image: "docker:stable",
ttyEnabled: true,
command: 'cat'
),
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
nfsVolume(mountPath: '/usr/local/apache-maven/repo', serverAddress: '10.0.0.103' , serverPath: '/opt/nfs/maven'),
],
)
{
node("jenkins-slave"){
// 第一步
stage('拉取代码'){
checkout([$class: 'GitSCM', branches: [[name: '${branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
}
// 第二步
stage('代码编译'){
//编译并安装公共工程
sh "mvn -f tensquare_common clean install"
}
// 第三步
stage('构建镜像，部署项目'){
//把选择的项目信息转为数组
def selectedProjects = "${project_name}".split(',')
for(int i=0;i<selectedProjects.size();i++){
//取出每个项目的名称和端口
def currentProject = selectedProjects[i];
//项目名称
def currentProjectName = currentProject.split('@')[0]
//项目启动端口
def currentProjectPort = currentProject.split('@')[1]
//定义镜像名称
def imageName = "${currentProjectName}:${tag}"
//编译，构建本地镜像
sh "mvn -f ${currentProjectName} clean package dockerfile:build"
container('docker') {
//给镜像打标签
sh "docker tag ${imageName} ${harbor_url}/${harbor_project_name}/${imageName}"
//登录Harbor，并上传镜像
withCredentials([usernamePassword(credentialsId: "${harbor_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
//登录
sh "docker login -u ${username} -p ${password} ${harbor_url}"
//上传镜像
sh "docker push ${harbor_url}/${harbor_project_name}/${imageName}"
}
//删除本地镜像
sh "docker rmi -f ${imageName}"
sh "docker rmi -f ${harbor_url}/${harbor_project_name}/${imageName}"
}
def deploy_image_name = "${harbor_url}/${harbor_project_name}/${imageName}"
//部署到K8S
sh """
sed -i 's#\$IMAGE_NAME#${deploy_image_name}#' ${currentProjectName}/deploy.yml
sed -i 's#\$SECRET_NAME#${secret_name}#' ${currentProjectName}/deploy.yml
"""
kubernetesDeploy configs: "${currentProjectName}/deploy.yml", kubeconfigId: "${k8s_auth}"
}
}
}
}
4）建立k8s认证凭证
# 将文件内容全部复制
[root@k8s-master ~]# cat /root/.kube/config
5）生成docker凭证
Docker凭证，用于Kubernetes到Docker私服拉取镜像
# 登录Harbor
[root@k8s-master ~]# docker login -u admin -p Harbor123456 10.0.0.101:85
# 生成docker凭证
[root@k8s-master ~]# kubectl create secret docker-registry registry-auth-secret --docker-server=10.0.0.101:85 --docker-username=admin --docker-password=Harbor12345 --docker-email=lawrence@qq.com
secret/registry-auth-secret created
# 查看密钥
[root@k8s-master ~]# kubectl get secret
NAME TYPE DATA AGE
default-token-kvdv8 kubernetes.io/service-account-token 3 8h
nfs-client-provisioner-token-gx7gf kubernetes.io/service-account-token 3 7h19m
registry-auth-secret kubernetes.io/dockerconfigjson 1 13s
6）每个项目建立deploy.xml
Eureka的deply.yml
---
apiVersion: v1
kind: Service
metadata:
name: eureka
labels:
app: eureka
spec:
type: NodePort
ports:
- port: 10086
name: eureka
targetPort: 10086
selector:
app: eureka
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: eureka
spec:
serviceName: "eureka"
replicas: 2
selector:
matchLabels:
app: eureka
template:
metadata:
labels:
app: eureka
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: eureka
image: $IMAGE_NAME
ports:
- containerPort: 10086
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: EUREKA_SERVER
value: "http://eureka-0.eureka:10086/eureka/,http://eureka-1.eureka:10086/eureka/"
- name: EUREKA_INSTANCE_HOSTNAME
value: ${MY_POD_NAME}.eureka
podManagementPolicy: "Parallel"
其他项目的deploy.yml主要把名字和端口修改：
网关服务：
---
apiVersion: v1
kind: Service
metadata:
name: zuul
labels:
app: zuul
spec:
type: NodePort
ports:
- port: 10020
name: zuul
targetPort: 10020
selector:
app: zuul
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zuul
spec:
serviceName: "zuul"
replicas: 2
selector:
matchLabels:
app: zuul
template:
metadata:
labels:
app: zuul
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: zuul
image: $IMAGE_NAME
ports:
- containerPort: 10020
podManagementPolicy: "Parallel"
权限管理：不要写成admin_service，会报错
---
apiVersion: v1
kind: Service
metadata:
name: admin
labels:
app: admin
spec:
type: NodePort
ports:
- port: 9001
name: admin
targetPort: 9001
selector:
app: admin
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: admin
spec:
serviceName: "admin"
replicas: 2
selector:
matchLabels:
app: admin
template:
metadata:
labels:
app: admin
spec:
imagePullSecrets:
- name: $SECRET_NAME
containers:
- name: admin
image: $IMAGE_NAME
ports:
- containerPort: 9001
podManagementPolicy: "Parallel"
7）项目构建后，查看服务创建情况
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
admin-0 1/1 Running 0 5m26s
admin-1 1/1 Running 0 5m26s
eureka-0 1/1 Running 0 34m
eureka-1 1/1 Running 0 34m
gathering-0 1/1 Running 0 15s
gathering-1 1/1 Running 0 15s
nfs-client-provisioner-989bf44d-gtn4k 1/1 Running 0 7h58m
zuul-0 1/1 Running 0 22m
zuul-1 1/1 Running 0 22m
[root@k8s-master ~]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
admin NodePort 10.1.59.3 <none> 9001:30178/TCP 5m30s
eureka NodePort 10.1.243.67 <none> 10086:32625/TCP 34m
gathering NodePort 10.1.193.246 <none> 9002:31407/TCP 20s
kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 9h
zuul NodePort 10.1.37.187 <none> 10020:31959/TCP 22m
作者：Lawrence
-------------------------------------------
个性签名：独学而无友，则孤陋而寡闻。做一个灵魂有趣的人！
扫描上面二维码关注我
如果你真心觉得文章写得不错，而且对你有所帮助，那就不妨帮忙“推荐"一下，您的“推荐”和”打赏“将是我最大的写作动力！
本文版权归作者所有，欢迎转载，但未经作者同意必须保留此段声明，且在文章页面明显位置给出原文连接.
