9.2 k8s结合Jenkins与gitlab实现代码升级与回滚
1.部署Jenkins
1.1 安装jdk
# apt 安装jdk11
apt install openjdk-11-jdk
# 查看
root@jenkins:~# java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
1.2 下载安装包
wget https://mirrors.tuna.tsinghua.edu.cn/jenkins/debian-stable/jenkins_2.303.2_all.deb
1.3 安装
# 安装依赖包
apk install daemon
# 安装jenkins
dpkg -i jenkins_2.303.2_all.deb
1.4 配置jenkins
# dpkg 安装jenkins后会启动服务,先停止
systemctl stop jenkins
# 更改jenkins的配置文件
root@jenkins:~# vim /etc/default/jenkins
# 开启自动触发的钩子的功能
JAVA_ARGS="-Djava.awt.headless=true -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true"
JAVA_ARGS="-Xmx4096m"
# make jenkins listen on IPv4 address
#JAVA_ARGS="-Djava.net.preferIPv4Stack=true"
PIDFILE=/var/run/$NAME/$NAME.pid
# user and group to be invoked as (default to jenkins)
JENKINS_USER=root
JENKINS_GROUP=root
# 再启动jenkins
root@jenkins:~# systemctl start jenkins.service
1.5 使用浏览器访问jenkins并配置
查看初始化密码
root@jenkins:~# cat /var/lib/jenkins/secrets/initialAdminPassword
89806b90623d4e0ebc8141db781b5790
2.部署gitlab
2.1 下载安装包
root@gitlab:~# wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/ubuntu/pool/focal/main/g/gitlab-ce/gitlab-ce_14.4.0-ce.0_amd64.deb
2.2 安装
root@gitlab:~# dpkg -i gitlab-ce_14.4.0-ce.0_amd64.deb
2.3 修改配置文件并重新加载配置
# 修改配置文件
root@gitlab:~# vim /etc/gitlab/gitlab.rb
external_url 'http://192.168.2.25'
# 重新加载配置
root@gitlab:~# gitlab-ctl reconfigure
...
Running handlers:
Running handlers complete
Chef Infra Client finished, 568/1519 resources updated in 06 minutes 30 seconds
Notes:
Default admin account has been configured with following details:
Username: root
Password: You didn't opt-in to print initial root password to STDOUT.
Password stored to /etc/gitlab/initial_root_password. This file will be cleaned up in first reconfigure run after 24 hours.
NOTE: Because these credentials might be present in your log files in plain text, it is highly recommended to reset the password following https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password.
gitlab Reconfigured!
2.4 获取root账号的初始密码
root@gitlab:~# grep ^Password /etc/gitlab/initial_root_password
Password: 20hTSLKLLe9mpeOq7pv+YU2mIQ0zsE9qOoWlo8h/GrA=
2.5 使用初始密码登录并修改root的密码
浏览器使用初始密码登录后,可以修改密码。
右上角'administrator' -> Edit profile -> password -> 输入老密码和新密码 -> 修改后使用新密重新登录
2.6 配置中文支持
右上角'administrator' -> Preferences -> Localization(下面) -> Language -> 选择中文 -> Save changes -> 刷新页面后即为中文。
2.7 配置项目和增加普通用户配置
-
创建群组(项目)
左上角“菜单” -> 管理员 -> 新建群组 -> 输入群组名称,比如:yan-test -> 创建群组
-
新建用户
左上角“菜单” -> 管理员 -> 新建用户 -> 输入名称(eg:user1),用户名(eg:user1),电子邮件(eg:user1@123.com) -> 创建用户 -> 编辑 -> 设置密码 -> 保存修改
-
新建项目(即群组(项目)下的服务)
左上角“菜单” -> 管理员 -> 新建项目 -> 创建空白项目 -> 输入项目名称(eg:app1)->项目URL 选 yan-test -> 新建项目 -> 自述文件 ->编辑 -> 文件名改为 index.html -> 内容输入:<h1> yan-test/app1/index.html v1</h1> ->提交信息输入:Update 2021.10.27 17:18 index.html -> commit changes
-
将新建的用户(user1)加入到群组(yan-test)
左上角“菜单” -> 管理员 -> 最新群组 -> yan-tset -> 将用户加入群组 ->user1 owner -> 将用户加入群组
-
使用新建的用(user1)重新登录并修改密码
-
获取项目app1的clone URL
左上角“菜单” -> 项目 -> 您的项目 -> yan-test/app -> 克隆 -> 点击 使用http克隆 下面的复制按钮获取url
2.8 测试从gitlab中clone代码
# 安装git
root@jenkins:~# apt install git
# 测试gti clone下载
root@jenkins:~/app1# git clone http://192.168.2.25/yan-test/app1.git
正克隆到 'app1'...
Username for 'http://192.168.2.25': user1
Password for 'http://user1@192.168.2.25':
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
展开对象中: 100% (6/6), 455 字节 | 151.00 KiB/s, 完成.
root@jenkins:~/app1# ls
app1
root@jenkins:~/app1# cd app1/
root@jenkins:~/app1/app1# ls
index.html
root@jenkins:~/app1/app1# cat index.html
<h1>yan-test/app1/index.html v1</h1>
root@jenkins:~/app1/app1#
# 测试git 上传提交
root@jenkins:~/app1/app1# git config --global user.email 'user1@123.com'
root@jenkins:~/app1/app1# git config --global user.name "user1"
root@jenkins:~/app1/app1# git commit -m 'update v2'
[main 1bde6f3] update v2
1 file changed, 1 insertion(+)
root@jenkins:~/app1/app1# git push
Username for 'http://192.168.2.25': user1
Password for 'http://user1@192.168.2.25':
枚举对象中: 5, 完成.
对象计数中: 100% (5/5), 完成.
使用 4 个线程进行压缩
压缩对象中: 100% (2/2), 完成.
写入对象中: 100% (3/3), 270 字节 | 270.00 KiB/s, 完成.
总共 3 (差异 0),复用 0 (差异 0)
To http://192.168.2.25/yan-test/app1.git
f4e9e94..1bde6f3 main -> main
通过web域名查看,代码已经更新至v2版本
2.9 配置jenkins免密访问gitlab
在jenkins server服务器上生成ssh密钥对
root@jenkins:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:NhUY6Ss/SnoRb7OSvJ2PTPCe4aeifskeuuvWQ9YTu8k root@jenkins
The key's randomart image is:
+---[RSA 3072]----+
| .+. |
| o . |
| . . |
| . o. |
| .+S+ |
| =+O. |
| *oO+* |
| .+%BE=. |
| +OB+*X=. |
+----[SHA256]-----+
查看公钥
root@jenkins:~# cat /root/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr+fxPl8b/0SZHhXo/5+UMixkRniJCBYqDP2AbEfzf93s8txCv50vRJ+G56pyei9b3vtgmT3ryvIjAQnqto0gP/0OXppS0MhfYVruWX7g+l+llRWiyw1oDaj14dlQDILlNNm2r8jMOyFV7YnhFv+OmM9VCsSUJw8L7yYQ6YeL6feT+YkLxnfoGvN46LbRdA7rGwO60taKCIA2PF1hNnJstWNn1CLyMNAALoqDjtfbFpbktooduz1WCZg47uSs9Pyw6oklNqMqSJO9kBaPo5lb12RC8Q1vA5pAV3X5qPylqqcaAgZ+KQnpYHxMhozG6EbykcfPL+NvUAH9uWvXRUFDLoLSfNth5509ujipds+Y3inL9ZoIZE5+frLqkKWgQn/s/NEXobT8D/adNNFkOSmfLmnAvuLpQ3VZKxh9nSd06Zz4QCUpQPpbVNHnkboYST3MS+M6MSP95wb5bEHprAfBLONnoS+ITHCnbB3xaOKKO8GvbgjpFZV8WqAaQ+L5hkLk= root@jenkins
复制公钥到gitlab
右上角‘administrator’ -> 偏好设置 -> SSH密钥 -> 复制密码内容进去 ->添加密钥
免密clone测试
# 删除之前使用http克隆的app1
root@jenkins:~# rm -rf app1/
# 使用app的git url克隆测试
root@jenkins:~# git clone git@192.168.2.25:yan-test/app1.git
正克隆到 'app1'...
The authenticity of host '192.168.2.25 (192.168.2.25)' can't be established.
ECDSA key fingerprint is SHA256:5MpImDTozrfwVxOhNf/C/vkMYBdLwoQlr3PK5qftwA4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.2.25' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 0
接收对象中: 100% (9/9), 完成.
root@jenkins:~#
3.实现代码部署与回滚
- 部署流程:
jenkins传参并调用脚本 -> 根据不同的参数从gitlab上获取不同的代码并编译打包 -> 将编译打包的代码文件复制到项目的镜像制作文件夹 -> 制作镜像时加入最新的代码包并上传到本地harbor -> 更改项目yaml文件中的镜像名 -> 应用yaml文件更新镜像 - 回滚流程:
jenkins传参并调用脚本 -> 脚本获取‘回滚’参数 -> 使用"kubect undo ..."回滚到上一个版本的镜像
3.1 gitlab上为项目创建分支
菜单 -> 您的项目-> 选择'yan-test/app'项目 ->点击 + ,新建分支 ->输入名称(eg:develop)
3.2 jenkins配置
3.2.1 新建一个‘自由风格’的项目
3.2.2 增加“2个构建参数” 配置(This project is parameterized)
3.2.3 增加“构建 执行shell”配置
shell脚本
root@jenkins:/data/scripts# cat yan-test-tomcat-app1.sh
#!/bin/bash
#Author: ZhangShiJie
#Date: 2018-10-24
#Version: v1
#记录脚本开始执行时间
starttime=`date +'%Y-%m-%d %H:%M:%S'`
#变量
SHELL_DIR="/root/scripts"
SHELL_NAME="$0"
K8S_CONTROLLER1="192.168.2.10"
#K8S_CONTROLLER2="172.31.7.102"
DATE=`date +%Y-%m-%d_%H_%M_%S`
METHOD=$1
Branch=$2
if test -z $Branch;then
Branch=develop
fi
function Code_Clone(){
Git_URL="git@192.168.2.25:yan-test/app1.git"
DIR_NAME=`echo ${Git_URL} |awk -F "/" '{print $2}' | awk -F "." '{print $1}'`
DATA_DIR="/data/gitdata/yan-test/"
Git_Dir="${DATA_DIR}/${DIR_NAME}"
cd ${DATA_DIR} && echo "即将清空上一版本代码并获取当前分支最新代码" && sleep 1 && rm -rf ${DIR_NAME}
echo "-------------------------------------------------------"
echo "即将开始从分支${Branch} 获取代码" && sleep 1
git clone -b ${Branch} ${Git_URL}
echo "分支${Branch} 克隆完成,即将进行代码编译!"
echo "-------------------------------------------------------" && sleep 1
#cd ${Git_Dir} && mvn clean package
#echo "代码编译完成,即将开始将IP地址等信息替换为测试环境"
#####################################################
sleep 1
cd ${Git_Dir}
tar czf ${DIR_NAME}.tar.gz ./*
}
#将打包好的压缩文件拷贝到k8s 控制端服务器
function Copy_File(){
echo "压缩文件打包完成,即将拷贝到k8s 控制端服务器${K8S_CONTROLLER1}" && sleep 1
scp ${Git_Dir}/${DIR_NAME}.tar.gz root@${K8S_CONTROLLER1}:/root/k8s-yaml/yun/backend/image
echo "压缩文件拷贝完成,服务器${K8S_CONTROLLER1}即将开始制作Docker 镜像!"
echo "-------------------------------------------------------" && sleep 1
}
#到控制端执行脚本制作并上传镜像
function Make_Image(){
echo "开始制作Docker镜像并上传到Harbor服务器" && sleep 1
ssh root@${K8S_CONTROLLER1} "cd /root/k8s-yaml/yun/backend/image && bash build-command.sh ${DATE}"
echo "Docker镜像制作完成并已经上传到harbor服务器"
echo "-------------------------------------------------------" && sleep 1
}
#到控制端更新k8s yaml文件中的镜像版本号,从而保持yaml文件中的镜像版本号和k8s中版本号一致
function Update_k8s_yaml(){
echo "即将更新k8s yaml文件中镜像版本" && sleep 1
ssh root@${K8S_CONTROLLER1} \
"cd /root/k8s-yaml/yun/backend && \
sed -i -e 's#image: 192.*#image: 192.168.1.110/yun/tomcat-app1:${DATE}#g' tomcat-app1.yaml \
tomcat-app1.yaml"
echo "k8s yaml文件镜像版本更新完成,即将开始更新容器中镜像版本"
echo "-------------------------------------------------------" && sleep 1
}
#到控制端更新k8s中容器的版本号,有两种更新办法,一是指定镜像版本更新,二是apply执行修改过的yaml文件
function Update_k8s_container(){
#第一种方法
#ssh root@${K8S_CONTROLLER1} "kubectl set image deployment/tomcat-deploy tomcat-ct=192.168.1.110/yun/tomcat-app1:${DATE} -n yan-test"
#第二种方法,推荐使用第一种
ssh root@${K8S_CONTROLLER1} "cd /root/k8s-yaml/yun/backend && kubectl apply -f tomcat-app1.yaml --record"
echo "k8s 镜像更新完成" && sleep 1
echo "当前业务镜像版本: harbor.magedu.net/magedu/tomcat-app1:${DATE}"
#计算脚本累计执行时间,如果不需要的话可以去掉下面四行
endtime=`date +'%Y-%m-%d %H:%M:%S'`
start_seconds=$(date --date="$starttime" +%s);
end_seconds=$(date --date="$endtime" +%s);
echo "本次业务镜像更新总计耗时:"$((end_seconds-start_seconds))"s"
}
#基于k8s 内置版本管理回滚到上一个版本
function rollback_last_version(){
echo "即将回滚之上一个版本"
ssh root@${K8S_CONTROLLER1} "kubectl rollout undo deployment/tomcat-deploy -n yan-test"
sleep 1
echo "已执行回滚至上一个版本"
}
#使用帮助
usage(){
echo "部署使用方法为 ${SHELL_DIR}/${SHELL_NAME} deploy "
echo "回滚到上一版本使用方法为 ${SHELL_DIR}/${SHELL_NAME} rollback_last_version"
}
#主函数
main(){
case ${METHOD} in
deploy)
Code_Clone;
Copy_File;
Make_Image;
Update_k8s_yaml;
Update_k8s_container;
;;
rollback_last_version)
rollback_last_version;
;;
*)
usage;
esac;
}
main $1 $2
3.3 发布、回滚测试
3.3.1 发布测试
点击“build with Parameters”开始构建 ->选择参数 -> 开始构建 -> 查看“控制台输出”
控制台输出信息如下,如有错误,根据相关提示信息对脚本进行修改。
Started by user admin
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/tomcat-app1
[tomcat-app1] $ /bin/sh -xe /tmp/jenkins3073480597549287762.sh
+ bash /data/scripts/yan-test-tomcat-app1.sh deploy main
即将清空上一版本代码并获取当前分支最新代码
-------------------------------------------------------
即将开始从分支main 获取代码
正克隆到 'app1'...
分支main 克隆完成,即将进行代码编译!
-------------------------------------------------------
压缩文件打包完成,即将拷贝到k8s 控制端服务器192.168.2.10
压缩文件拷贝完成,服务器192.168.2.10即将开始制作Docker 镜像!
-------------------------------------------------------
开始制作Docker镜像并上传到Harbor服务器
Sending build context to Docker daemon 205.9MB
Step 1/15 : FROM 192.168.1.110/base/alpine:v20211010-1704
---> 14119a10abf4
Step 2/15 : LABEL maintainer="yanql<yanqianling@faxuan.net>"
---> Using cache
---> 36b524e2e2c9
Step 3/15 : ADD jdk-8u192-linux-x64.tar.gz /usr/local/
---> Using cache
---> 182fcf2b50ec
Step 4/15 : ADD apache-tomcat-8.5.70.tar.gz /usr/local/
---> Using cache
---> 0fb49d57c606
Step 5/15 : ADD webapps.tgz /opt/
---> Using cache
---> 93bb8d40b8c6
Step 6/15 : ADD app1.tar.gz /opt/webapps/app1/
---> fba96d9ba096
Step 7/15 : COPY catalina.sh /usr/local/apache-tomcat-8.5.70/bin/catalina.sh
---> 725b7a04fbf0
Step 8/15 : COPY glibc-2.29-r0.apk /
---> d0cfbca2e287
Step 9/15 : COPY server.xml /usr/local/apache-tomcat-8.5.70/conf/
---> 13497d92ec16
Step 10/15 : COPY tomcat-users.xml /usr/local/apache-tomcat-8.5.70/conf/
---> 21d729f1895b
Step 11/15 : RUN set -x && addgroup -g 101 -S nginx && adduser -S -D -H -u 101 -h /var/cache/nginx -s /sbin/nologin -G nginx -g nginx nginx && chown -R nginx:nginx /opt/webapps /usr/local/apache-tomcat-8.5.70 && sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && apk add glibc-2.29-r0.apk && rm -rf /usr/local/apache-tomcat-8.5.70/webapps
---> Running in cb3f7cddbbf6
[91m+ addgroup -g 101 -S nginx
[0m[91m+ adduser -S -D -H -u 101 -h /var/cache/nginx -s /sbin/nologin -G nginx -g nginx nginx
[0m[91m+ chown -R nginx:nginx /opt/webapps /usr/local/apache-tomcat-8.5.70
[0m[91m+ sed -i s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g /etc/apk/repositories
[0m[91m+ wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
[0m[91m+ apk add glibc-2.29-r0.apk
[0mfetch https://mirrors.ustc.edu.cn/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
fetch https://mirrors.ustc.edu.cn/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
(1/1) Installing glibc (2.29-r0)
OK: 10 MiB in 15 packages
[91m+ rm -rf /usr/local/apache-tomcat-8.5.70/webapps
[0mRemoving intermediate container cb3f7cddbbf6
---> 55a5a0260538
Step 12/15 : ENV JAVA_HOME /usr/local/jdk1.8.0_192
---> Running in 86eb45829ee7
Removing intermediate container 86eb45829ee7
---> 6a5bbaac4dcc
Step 13/15 : ENV PATH ${PATH}:${JAVA_HOME}/bin
---> Running in 1e67c1e72a76
Removing intermediate container 1e67c1e72a76
---> fdd4cfbceb57
Step 14/15 : EXPOSE 8080
---> Running in 9e78089af6b8
Removing intermediate container 9e78089af6b8
---> 57a71fac9eb9
Step 15/15 : CMD ["/usr/local/apache-tomcat-8.5.70/bin/catalina.sh", "run"]
---> Running in b6446f282451
Removing intermediate container b6446f282451
---> 43bafa3a6989
Successfully built 43bafa3a6989
Successfully tagged 192.168.1.110/yun/tomcat-app1:2021-10-28_13_01_59
The push refers to repository [192.168.1.110/yun/tomcat-app1]
8eef12e72f9c: Preparing
12288c868853: Preparing
13eb0d1777fd: Preparing
055bbeaa30a9: Preparing
33e602b13ef0: Preparing
47e2d7a67650: Preparing
dc823d65138e: Preparing
4ba66f116ecf: Preparing
6398b414e515: Preparing
e2eb06d8af82: Preparing
4ba66f116ecf: Waiting
6398b414e515: Waiting
47e2d7a67650: Waiting
dc823d65138e: Waiting
e2eb06d8af82: Waiting
055bbeaa30a9: Layer already exists
12288c868853: Pushed
33e602b13ef0: Pushed
dc823d65138e: Layer already exists
13eb0d1777fd: Pushed
4ba66f116ecf: Layer already exists
6398b414e515: Layer already exists
47e2d7a67650: Pushed
e2eb06d8af82: Layer already exists
8eef12e72f9c: Pushed
2021-10-28_13_01_59: digest: sha256:03f3b935a746b99b3c38c2868d6448dbd5b233822b8a033eef5f6ece7ce63372 size: 2418
Docker镜像制作完成并已经上传到harbor服务器
-------------------------------------------------------
即将更新k8s yaml文件中镜像版本
k8s yaml文件镜像版本更新完成,即将开始更新容器中镜像版本
-------------------------------------------------------
service/tomcat-svc unchanged
deployment.apps/tomcat-deploy configured
k8s 镜像更新完成
当前业务镜像版本: harbor.magedu.net/magedu/tomcat-app1:2021-10-28_13_01_59
本次业务镜像更新总计耗时:42s
Finished: SUCCESS
访问项目页面测试
#curl '192.168.2.17:38080/app1/index.html'
<h1>yan-test/app1/index.html v1</h1>
3.3.2 更新测试
将项目apps/index.html的内容镜像更新,比如:
在jenkins中重新“build with Parameters”,通过查看“控制台输出”,等构建成功后,使用浏览器测试访问app1/index.html