kubernetes gitlab runner java maven ci/cd 整体方案示例
基于gitlab runner 的did(docker in docker ) ci/cd k8s方案
首先，jenkins很强大，尤其是各种插件的支持，但实际个人工作中，用到的并不多，早期大型项目布署负载各种脚本和远程调用，目前所有项目和k8s深耦合，已经拆解为各种云服务，jenkins的大部分功能用不到
其次，这只是一种可行的方案，并不是最优的方案，不同阶段也都有再调整和优化的空间
最后，对个人的需求，jenkins过于复杂，gitlab-runner足可胜任，对研发人员更透明友好，学习应用成本更低。
CI/CD集成k8s服务,对各种文件抽像的只有三步（部分语言生态只需要一步）
- 1 编译环境加载依赖编译可执行文件
- 2 执行环境+可执行文件打包为docker image并上传至docker hub
- 3 执行k8s布署，启动服务
k8s布署k8s gitlab-runner 服务
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: gitlab-runner-java
namespace: common
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: gitlab-runner-java
serviceName: gitlab-runner-java
template:
metadata:
creationTimestamp: null
labels:
app: gitlab-runner-java
name: gitlab-runner-java
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- server-svc-2
containers:
- image: gitlab/gitlab-runner:v12.4.1
imagePullPolicy: IfNotPresent
name: gitlab-runner-java
resources:
limits:
cpu: "4"
memory: 4Gi
requests:
cpu: "2"
memory: 1Gi
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /var/run/docker.sock
name: docker-sock
readOnly: true
- mountPath: /etc/gitlab-runner
name: config
- mountPath: /cache
name: cache
- mountPath: /usr/share/maven/ref/repository
name: maven
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /var/run/docker.sock
type: ""
name: docker-sock
- hostPath:
path: /data1/pv/gitlab-runner-config/gitlab-runner-java
type: ""
name: config
- hostPath:
path: /data1/pv/gitlab-runner-cache/gitlab-runner-java
type: ""
name: cache
- hostPath:
path: /data2/pv/gitlab-runner-java-maven
type: ""
name: maven
首先布署gitlab-runner节点，使用did方案,did要直接访问 /var/run/docker.sock因此需要较高的权限
个人比较熟悉k8s,k8s权限提升使用
securityContext:
runAsUser: 0
另外需挂载几项目录，
- /var/run/docker.sock:/var/run/docker.sock 自不必说,did 必备
- :/etc/gitlab-runner 默认容器内配置文件路径为/etc/gitlab-runner/config.toml，挂载方便维护
- /cache 自不必说
- :/usr/share/maven/ref/repository maven的默认的本地缓存repository路径
需要补充的是，只有/var/run/docker.sock是did必须 其他三项是为了可以从k8s访问本地挂载目录提供的，方便在没有宿主机权限下，通过k8s pod访问目录
docker ps |grep java
759e94f0dafd db5c21d7a3e4 "/usr/bin/dumb-init …" 6 weeks ago Up 6 weeks k8s_gitlab-runner-java_gitlab-runner-java-0_common_9112c414-cef3-4801-a234-4c8edb965e82_0
6bda34787976 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 "/pause" 6 weeks ago Up 6 weeks k8s_POD_gitlab-runner-java-0_common_9112c414-cef3-4801-a234-4c8edb965e82_0
目前为不同语言构建了不同的pod,以java的为例(java的本身较为复杂，maven打包为jar,再将jar以java包装为镜像)
gitlab-runner gitlab-runner-java-0 1/1 Running 0 47d
gitlab-runner gitlab-runner-node-0 1/1 Running 0 47d
gitlab-runner gitlab-runner-python-0 1/1 Running 0 47d
gitlab-runner gitlab-runner-golang-0 1/1 Running 0 47d
进入gitlab后台，获取项目的token
进入容器以token 注册gitlab-runner
---
root@gitlab-runner-java-0:/# gitlab-runner register -n \
> --url https://[abc.gitlab.com] \
> --registration-token [your_project_token] \
> --tag-list java,prod,preview,test \
> --executor docker \
> --description "java" \
> --cache-dir /data1/pv/gitlab-runner-cache/gitlab-runner-java/cache-dir \
> --docker-cache-dir /data1/pv/gitlab-runner-cache/gitlab-runner-java/docker-cache-dir \
> --docker-image "docker:18.06.3" \
> --docker-pull-policy "if-not-present" \
> --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
> --docker-volumes /root/.docker/config.json:/root/.docker/config.json \
> --docker-volumes /data1/pv/gitlab-runner-cache/gitlab-runner-java:/cache \
> --docker-volumes /data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository
Runtime platform arch=amd64 os=linux pid=94 revision=05161b14 version=12.4.1
Running in system-mode.
Registering runner... succeeded runner=joTBdSTv
重点注意--docker-volumes的几项参数，因为是did的执行方式，外部目录尤其是(/root/.m2/repository)需要在maven容器持久化，k8s yaml mount的路径只是查看需要，gitlab-runner注册时的docker-volumes才是真正在不同任务间共享数据的必须
例如--docker-volumes /root/.docker/config.json:/root/.docker/config.json 宿主机docker login的授权信息，可以共享使用
--docker-volumes /data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository 对maven容器，不同项目共同挂载宿主机目录，减少jar包的重复下载
注册完后的内容如
root@gitlab-runner-java-0:/# cat /etc/gitlab-runner/config.toml
concurrent = 8
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "java"
url = "https://[abc.123.com]"
token = "[your token]"
executor = "docker"
cache_dir = "/data1/pv/gitlab-runner-cache/gitlab-runner-java/cache-dir"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "docker:18.06.3"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/root/.docker/config.json:/root/.docker/config.json", "/data1/pv/gitlab-runner-cache/gitlab-runner-java:/cache", "/data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository"]
cache_dir = "/data1/pv/gitlab-runner-cache/gitlab-runner-java/docker-cache-dir"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
重点的项目配置文件，具体项目结构见https://github.com/cclient/hanlp-remote-dict
.
├── Dockerfile --打包docker镜像配置
├── deploy --镜像打包成功，更新上线至k8s配置
│ ├── deployment_preview.yaml
│ ├── deployment_prod.yaml
│ └── deployment_test.yaml
├── .gitlab-ci.yml --gitlab-runner ci/cd配置
├── pom.xml --java项目依赖配置
.gitlab-ci.yml
variables:
DOCKER_API_VERSION: '1.38'
DOCKER_HUB: "[docker hub/registry]"
DOCKER_USERNAME: "[your docker user]"
DOCKER_PASSWORD: "[your docker passwd]"
PROJECT_NAME: $CI_PROJECT_NAME
PROJECT_VERSION: $CI_COMMIT_TAG
K8S_API_URL: "https://[k8s api host]:6443"
K8S_ACCESS_TOKEN: "[your k8s token]"
K8S_DEPLOY_BASE_NAME: "hanlp-remote-dict"
CONTAINER_IMAGE: $DOCKER_HUB/$CI_PROJECT_NAME
image: docker:18.06.3
stages:
- prepare-env
- maven-package
- docker-build
- k8s-deploy
### 根据git branch/tag匹配 cicd上线环境，添加相关环境变量
prepare-env-develop:
stage: prepare-env
tags:
- java
only:
- develop
before_script:
- mkdir -p ./sartifacts/
script:
- echo BUILD_ENV="test" >> ./sartifacts/version
- echo BUILD_IMAGE_VERSION="test" >> ./sartifacts/version
artifacts:
name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
paths:
- ./sartifacts/*
expire_in: 2 day
prepare-env-preview:
stage: prepare-env
tags:
- java
only:
- /^release\/.*$/
- master
before_script:
- echo docker login hub.intra.github.com -u "'$DOCKER_USERNAME'" -p $DOCKER_PASSWORD
- mkdir -p ./sartifacts/
script:
- echo BUILD_ENV="preview" >> ./sartifacts/version
- echo BUILD_IMAGE_VERSION="preview" >> ./sartifacts/version
artifacts:
name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
paths:
- ./sartifacts/*
expire_in: 2 day
prepare-env-production:
stage: prepare-env
tags:
- java
only:
- tags
before_script:
- mkdir -p ./sartifacts/
script:
- echo BUILD_ENV="prod" >> ./sartifacts/version
- echo BUILD_IMAGE_VERSION=$CI_COMMIT_TAG >> ./sartifacts/version
artifacts:
name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
paths:
- ./sartifacts/*
expire_in: 2 day
### 以下为通用部分
#### mvn 打包java项目，生成jar包，并将jar包添加至./sartifacts/ 传至下一级
maven-package:
image: maven:3.6.3-adoptopenjdk-8
stage: maven-package
tags:
- java
only:
- tags
- /^release\/.*$/
- master
- develop
before_script:
# source加载./sartifacts/version的环境变量
- source ./sartifacts/version
# echo只是调试用，打印gitlab-runner结合git的变量
- echo 'CI_CONFIG_PATH' $CI_CONFIG_PATH
- echo 'CI_COMMIT_REF_NAME' $CI_COMMIT_REF_NAME
- echo 'CI_COMMIT_BRANCH' $CI_COMMIT_BRANCH
- echo 'CI_COMMIT_TAG' $CI_COMMIT_TAG
- echo 'CI_COMMIT_TITLE' $CI_COMMIT_TITLE
- echo 'CI_ENVIRONMENT_NAME' $CI_ENVIRONMENT_NAME
- echo 'CONTAINER_IMAGE' $CONTAINER_IMAGE
- echo 'ENV' $ENV
- echo 'MAVEN_OPTS' "$MAVEN_OPTS"
- echo 'BUILD_ENV' "$BUILD_ENV"
- echo 'BUILD_IMAGE_VERSION' "$BUILD_IMAGE_VERSION"
script:
- mvn clean package -Dmaven.test.skip=true -Dmaven.repo.local=/root/.m2/repository
- mv target ./sartifacts/
artifacts:
name: "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
paths:
- ./sartifacts/*
expire_in: 2 day
#### 获取上一级的sartifacts，提取target目录，打包image并上传
docker-build:
image: docker:18.06.3
stage: docker-build
tags:
- java
only:
- tags
- /^release\/.*$/
- master
- develop
before_script:
- pwd
- ls -alh
- source ./sartifacts/version
- mv ./sartifacts/target ./target
script:
- docker build -t $CONTAINER_IMAGE:$BUILD_IMAGE_VERSION ./
# - docker login hub.intra.github.com -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
- docker push $CONTAINER_IMAGE:$BUILD_IMAGE_VERSION
#### 执行k8s yaml 上线pod/deploy/sts等服务
k8s-deploy:
image: telefonica/kubectl:1.15.10
stage: k8s-deploy
tags:
- java
only:
- tags
- /^release\/.*$/
- master
- develop
before_script:
- source ./sartifacts/version
- echo K8S_DEPLOY_BASE_NAME-BUILD_ENV "$K8S_DEPLOY_BASE_NAME-$BUILD_ENV"
- echo CONTAINER_IMAGE "$CONTAINER_IMAGE"
script:
- kubectl --server="${K8S_API_URL}" --token="$K8S_ACCESS_TOKEN" --insecure-skip-tls-verify apply -f ./deploy/deployment_$BUILD_ENV.yaml -n default
- kubectl --server="${K8S_API_URL}" --token="$K8S_ACCESS_TOKEN" --insecure-skip-tls-verify set image deployment/$K8S_DEPLOY_BASE_NAME-$BUILD_ENV $K8S_DEPLOY_BASE_NAME-$BUILD_ENV=$CONTAINER_IMAGE:$BUILD_IMAGE_VERSION -n default --record
基本流程为
- 1 第一阶段 prepare-env 根据gitlab 提交和分枝信息，构造version文件(保存env),供后续阶段使用，目前实现了三项prepare-env-develop，prepare-env-preview，prepare-env-production，内容一样是写入version,为了保证这里不够精简，看了下gitlab-runner的rule，应该可以有更优雅的方案，但个人没有精力深入，有知道更好办法的，烦请告知。
- 2 第二阶段 maven-package did以maven:3.6.3-adoptopenjdk-8为底包打包项目，生成target目录(下有jar包)，并将targer整个打包为sartifacts传给下一阶段，大包的话，只用jar即可
- 3 第三阶段 docker-build dit以docker:18.06.3为底包，自动下载解压sartifacts，拷备出target至项目目标地址，根据Dockerfile(对该示例而言底包为openjdk:8u162-jre-slim-stretch) 生成image 并上传,
docker login hub.intra.github.com -u $DOCKER_USERNAME -p $DOCKER_PASSWORD这一步如果挂载了宿主机的docker 授权则不必要 - 4 第四阶段 k8s-deploy did 以telefonica/kubectl:1.15.10为底包,因需选择版本，个人用该镜像从k8s v1.15.0用到v1.18.5，目前使用正常，通过api 提交相应deploy.yaml至k8s api，这一阶段执行两部，kubectl apply -f deploy.yaml 提交上线，再kubectl set image 更新deploy内服务image的真实版本
docker file
FROM openjdk:8u162-jre-slim-stretch
WORKDIR /home/spring
ADD target/*.jar /home/spring/app.jar
deploy.yaml 以测试deploy为例
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: hanlp-remote-dict-test
name: hanlp-remote-dict-test
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: hanlp-remote-dict-test
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: hanlp-remote-dict-test
spec:
containers:
- image: cclient/arm64v8-kafka:test
command:
- java
args:
- -Xms1g
- -Xmx2g
- -XX:+UseG1GC
- -XX:G1ReservePercent=10
- -XX:+UseStringDeduplication
- -jar
- app.jar
imagePullPolicy: Always
name: hanlp-remote-dict-test
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 1Gi
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: hanlp-remote-dict-test
namespace: default
spec:
externalTrafficPolicy: Local
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: hanlp-remote-dict-test
sessionAffinity: None
type: NodePort
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hanlp-remote-dict-test
namespace: default
spec:
rules:
- host: hanlp-remote-dict-test.github.com
http:
paths:
- path: /
backend:
serviceName: hanlp-remote-dict-test
servicePort: 8080
执行成功样例
[session_server].listen_address not defined, session endpoints disabled builds=0
Checking for jobs... received job=196962 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
Job succeeded duration=21.427262629s job=196962 project=11190 runner=sdWbANrN
Checking for jobs... received job=196963 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
Job succeeded duration=2m3.362519176s job=196963 project=11190 runner=sdWbANrN
Checking for jobs... received job=196964 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
Job succeeded duration=4m8.502822318s job=196964 project=11190 runner=sdWbANrN
Checking for jobs... received job=196965 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
其他
1 目前spring boot 构造为一个大包，docker image上传下载io开销较大，可调整为先下载准备了依赖，再打小包
2 个人没有使用https://github.com/fabric8io/docker-maven-plugin等mvn 组件，主要原因是要支持多语言，方案不同，但共用通用的ci/cd抽像，对其他语言而言，可能不存在相同定位的包管理插件，docker cmd 更通用，对环境的依赖也小
3 test,preview,prod 不同上线分枝可以提交不同的 application.properties/application.yaml配置
限于篇幅太长，最后test,preview,prod对应git 分枝管理，以后再弹独写一篇
End
