03: 构建持续集成环境
1.1 实验环境介绍
1、部署服务介绍（centos 7.3）
1. k8s平台(部署Jenkins)： 192.168.56.11、192.168.56.12、192.168.56.13
2. Git / Harbor： 192.168.56.14
2、准备工作
1）对项目的理解
1. 单体架构？微服务？
2. 怎么部署？
3. 启动是否有依赖？
2）部署到k8s平台怎么个流程？
1. 制作镜像
2. 容器放到Pod
3. 控制器管理Pod
4. 暴露应用
5. 对外发布应用
6. 日志管理/监控
3）不同环境区分配置文件
1. configmap
2. entrypoint.sh
3. Apollo， Disconf（统一配置中心：每次启动时会从配置中心拉取配置 ）
1.2 安装所需服务
1、安装Harbor（192.168.56.14）
注：Harbor必须要先按照docker
# 1）安装依赖包 yum install -y yum-utils device-mapper-persistent-data lvm2 # 2）添加Docker软件包源(否则doker安装的不是新版本) yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo # 3）安装Docker CE yum install -y docker-ce # 4）启动Docker服务并设置开机启动 systemctl start docker systemctl enable docker # 5）安装docker compose curl -L https://github.com/docker/compose/releases/download/1.24.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose docker-compose --version # 测试docker-compose是否安装好
'''安装Harbor镜像仓库''' # 1、下载离线安装包 https://github.com/goharbor/harbor/releases # 2、解压并配置访问地址''' tar zxvf harbor-offline-installer-v1.8.1.tgz cd /aaa/harbor vi harbor.yml ''' hostname = 192.168.56.14 harbor_admin_password = 123456 ''' # 3、准备配置 ./prepare # 4、导入镜像并启动 ./install.sh # 5、查看容器状态 docker-compose ps
安装完成后可以登录Harbor：http://192.168.56.14
2、Git代码版本仓库
'''在192.168.56.14安装git仓库 ''' # 1、安装Git yum -y install git # 2、创建Git用户并设置密码 useradd git passwd git # 3、创建仓库（下面三步git仓库已经创建完成） su - git # 切换到git用户 mkdir demo.git # 在git用户家目录创建镜像仓库文件夹 demo.git cd demo.git git --bare init # 初始化镜像仓库 '''在192.168.56.13中模拟拉取git代码进行测试 ''' # 4、模拟拉取并提交代码 yum -y install git git clone git@192.168.56.14:/home/git/demo.git touch test.py # 创建一个文件模拟代码提交 git add . git commit -m 'test' git push origin master # 5、配置客户端与Git服务器SSH免交互认证 ssh-keygen # 生成秘钥 ssh-copy-id git@192.168.56.14 # 将私钥拷贝到git服务器 ssh git@192.168.56.14 # 在192.168.56.13测试免密登录192.168.56.14
1.3 在Kubernetes中部署Jenkins
Jenkins官网地址：https://jenkins.io/zh/download/
参考此网址在k8s中部署Jenkins：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes
1、安装nfs服务
1）由于Jenkins安装文件在k8s集群中不确定，k8s中服务存储路径一般放到共享存储中
2）k8s一般用pv来实现共享存储，这里简化了，直接安装nfs服务
3）nfs-utils即是客户端的包也是客户端的支持，所以在需要挂载nfs的机器中都需要安装nfs-utils
4）在192.168.56.14创建nfs共享文件，在192.168.56.11~13中通过网络挂载
'''注：记得在192.168.56.14中启动harbor'''
[root@linux-node4 harbor]# cd /aaa/harbor
[root@linux-node4 harbor]# docker-compose up -d # 启动harbor
'''1、在192.168.56.14中安装nfs服务，将其/ifs/kubernetes文件夹暴露到网络中供k8s集群使用'''
[root@linux-node4 ~]# yum -y install nfs-utils
[root@linux-node4 ~]# mkdir -p /ifs/kubernetes # 需要暴露一个目录供k8s挂载
[root@linux-node4 ~]# vim /etc/exports # exports是nfs的共享目录配置文件
# 注：nfs将你本地的文件系统暴露在你的网络中，其他的节点可以访问它
'''
/ifs/kubernetes *(insecure,rw,async,no_root_squash) # 配置nfs挂载目录
'''
[root@linux-node4 ~]# systemctl restart nfs
'''2、nfs-utils即是服务端也是客户端，在192.168.56.11~13安装nfs-utils包'''
[root@linux-node1 ~]# yum -y install nfs-utils
[root@linux-node1 ~]# mount -t nfs 192.168.56.14:/ifs/kubernetes /mnt/ # 可以在192.168.56.11中测试挂载nfs
[root@linux-node1 ~]# umount /mnt/ # 取消挂载
2、在k8s集群中配置pv自动供给（在192.168.56.11 k8s master中配置）
1）创建pv自动供给的三个配置文件
[root@linux-node1 nfs-client]# cd /cicd/nfs-client/
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "true"
apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: nfs-client-provisioner spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner image: lizhenliang/nfs-client-provisioner:latest # 指定nfs客户端镜像 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: fuseim.pri/ifs - name: NFS_SERVER value: 192.168.56.14 # 这里必须设置为nfs服务端地址 - name: NFS_PATH value: /ifs/kubernetes # 这里必须指定nfs服务端共享的文件夹路径 volumes: - name: nfs-client-root nfs: server: 192.168.56.14 # 这里必须设置为nfs服务端地址 path: /ifs/kubernetes # 这里必须指定nfs服务端共享的文件夹路径
kind: ServiceAccount apiVersion: v1 metadata: name: nfs-client-provisioner --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: default roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner namespace: default # replace with namespace where provisioner is deployed roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io
2）利用上面的配置文件创建pv供给
[root@linux-node1 nfs-client]# cd /cicd/nfs-client/
[root@linux-node1 nfs-client]# kubectl create -f . exists # 创建pv供给
[root@linux-node1 nfs-client]# kubectl get pods # 查看是否已经创建
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5b688d47-hxbbd 1/1 Running 0 51s
3、在k8s平台中创建Jenkins
1）编写文件完成Jenkins创建
[root@linux-node1 jenkins]# cd /cicd/K8S-CI/jenkins # 进入文件夹创建以下文件
apiVersion: v1 kind: Service metadata: name: jenkins spec: selector: name: jenkins type: NodePort ports: - name: http port: 80 targetPort: 8080 protocol: TCP nodePort: 30006 - name: agent port: 50000 protocol: TCP
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: jenkins annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" kubernetes.io/tls-acme: "true" # 如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size nginx.ingress.kubernetes.io/proxy-body-size: 50m nginx.ingress.kubernetes.io/proxy-request-buffering: "off" # nginx-ingress controller版本小于 0.9.0.beta-18 的配置 ingress.kubernetes.io/ssl-redirect: "true" ingress.kubernetes.io/proxy-body-size: 50m ingress.kubernetes.io/proxy-request-buffering: "off" spec: rules: - host: jenkins.example.com http: paths: - path: / backend: serviceName: jenkins servicePort: 80
--- apiVersion: v1 # 创建名为jenkins的ServiceAccount kind: ServiceAccount metadata: name: jenkins --- kind: Role # 创建名为jenkins的Role，授予允许管理API组的资源Pod apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 # 将名为jenkins的Role绑定到名为jenkins的ServiceAccount kind: RoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins
--- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins --- kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins
apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: jenkins labels: name: jenkins spec: serviceName: jenkins replicas: 1 updateStrategy: type: RollingUpdate template: metadata: name: jenkins labels: name: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins:lts-alpine imagePullPolicy: Always ports: - containerPort: 8080 - containerPort: 50000 resources: limits: cpu: 1 memory: 1Gi requests: cpu: 0.5 memory: 500Mi env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 securityContext: fsGroup: 1000 volumeClaimTemplates: - metadata: name: jenkins-home spec: storageClassName: "managed-nfs-storage" accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi
2）利用上面文件完成Jenkins创建
[root@linux-node1 jenkins]# kubectl create -f . # 创建Jenkins
[root@linux-node1 jenkins]# kubectl get pods # 可以看到正在拉取镜像镜像创建
# 可以看到Jenkins的NodePort暴露地址30006，访问k8s集群中任意 ip:30006 可访问Jenkins主页
[root@linux-node1 jenkins]# kubectl get pods,svc # 查看状态
[root@linux-node1 jenkins]# kubectl describe pod jenkins-0 # jenkins-0 是上面查询到的pod信息
[root@linux-node1 jenkins]# kubectl logs jenkins-0 -f # 查看Jenkins日志，日志里有Jenkins初始密码
30e21a3ecebd4220bf30f1ec82269c50 # 这里是日志中查询到的初始密码
[root@linux-node1 jenkins]# kubectl get pods,svc # 只有当Jenkins中 READY和STATUS一致时才准备就绪
NAME READY STATUS RESTARTS AGE
pod/jenkins-0 1/1 running 9 43m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/jenkins NodePort 10.110.20.21 <none> 80:30006/TCP,50000:32444/TCP 43m
注：当上面两个状态正常后就可以在页面中访问了：http://192.168.56.13:30006/
3、在页面中初始化Jenkins
1）在页面中访问：http://192.168.56.13:30006 # node节点中的任意地址都可以访问Jenkins
2）解决安装Jenkins显示离线问题
'''1、浏览器访问''' http://192.168.56.13:30006/pluginManager/advanced '''2、将站点中默认的https修改为http''' http://updates.jenkins.io/update-center.json '''3、重启Jenkins''' 方法一：http://127.0.0.1:8080/restart 方法二：service jenkins restart
1.4 Jenkins在Kubernetes中动态创建代理
1、在页面中为Jenkins安装插件：git 和kubernetes
1. git然Jenkins拉取代码
2. kubernetes插件用于动态用kubernetes创建代理
2、安装插件(这种要连接国外网站无法成功安装)
1. 访问地址：http://192.168.56.13:30006/pluginManager/available
2. 在页面中勾选：kubernetes和Git
3. 然后选择：install and not restart
3、解决k8s中容器无法访问外网问题
'''1、检查宿主机 '''
[root@linux-node2 ~]# vi /etc/resolv.conf
'''
nameserver 114.114.114.114
nameserver 192.168.56.2
search lan
'''
'''2、配置网桥 '''
[root@linux-node2 ~]# vi /etc/resolv.conf
'''
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
'''
'''3、进入k8s容器内部docker容器配置dns '''
[root@linux-node2 ~]# docker exec -u root -it 95d9c57765de /bin/bash # 进入docker容器
bash-4.4# vi /etc/resolv.conf
'''
nameserver 10.96.0.10
nameserver 8.8.8.8
'''
4、配置kubernetes插件
111111111111
作者：学无止境
出处：https://www.cnblogs.com/xiaonq
生活不只是眼前的苟且，还有诗和远方。
