基于gitlab runner 的did(docker in docker ) ci/cd k8s方案一

首先，jenkins很强大，尤其是各种插件的支持，但实际个人工作中，用到的并不多，早期大型项目布署负载各种脚本和远程调用，目前所有项目和k8s深耦合，已经拆解为各种云服务，jenkins的大部分功能用不到

其次，这只是一种可行的方案，并不是最优的方案，不同阶段也都有再调整和优化的空间

最后，对个人的需求，jenkins过于复杂，gitlab-runner足可胜任，对研发人员更透明友好，学习应用成本更低。

CI/CD集成k8s服务,对各种文件抽像的只有三步（部分语言生态只需要一步）

  • 1 编译环境加载依赖编译可执行文件
  • 2 执行环境+可执行文件打包为docker image并上传至docker hub
  • 3 执行k8s布署，启动服务

k8s布署k8s gitlab-runner 服务

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitlab-runner-java
  namespace: common
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: gitlab-runner-java
  serviceName: gitlab-runner-java
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: gitlab-runner-java
      name: gitlab-runner-java
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - nb-sia-29
      containers:
      - image: gitlab/gitlab-runner:v12.4.1
        imagePullPolicy: IfNotPresent
        name: gitlab-runner-java
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "2"
            memory: 1Gi
        securityContext:
          runAsUser: 0
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock
          readOnly: true
        - mountPath: /etc/gitlab-runner
          name: config
        - mountPath: /cache
          name: cache
        - mountPath: /usr/share/maven/ref/repository
          name: maven
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/run/docker.sock
          type: ""
        name: docker-sock
      - hostPath:
          path: /data1/pv/gitlab-runner-config/gitlab-runner-java
          type: ""
        name: config
      - hostPath:
          path: /data1/pv/gitlab-runner-cache/gitlab-runner-java
          type: ""
        name: cache
      - hostPath:
          path: /data2/pv/gitlab-runner-java-maven
          type: ""
        name: maven

首先布署gitlab-runner节点，使用did方案,did要直接访问 /var/run/docker.sock因此需要较高的权限

个人比较熟悉k8s,k8s权限提升使用

        securityContext:
          runAsUser: 0

另外需挂载几项目录，

  • /var/run/docker.sock:/var/run/docker.sock 自不必说,did 必备
  • :/etc/gitlab-runner 默认容器内配置文件路径为/etc/gitlab-runner/config.toml，挂载方便维护
  • /cache 自不必说
  • :/usr/share/maven/ref/repository maven的默认的本地缓存repository路径

需要补充的是，只有/var/run/docker.sock是did必须 其他三项是为了可以从k8s访问本地挂载目录提供的，方便在没有宿主机权限下，通过k8s pod访问目录

[cuidapeng@nb-sia-29 ~]$ docker ps |grep java
759e94f0dafd        db5c21d7a3e4                                                    "/usr/bin/dumb-init …"   6 weeks ago         Up 6 weeks                                                                                                           k8s_gitlab-runner-java_gitlab-runner-java-0_common_9112c414-cef3-4801-a234-4c8edb965e82_0
6bda34787976        registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1   "/pause"                 6 weeks ago         Up 6 weeks                                                                                                           k8s_POD_gitlab-runner-java-0_common_9112c414-cef3-4801-a234-4c8edb965e82_0

目前为不同语言构建了不同的pod,以java的为例(java的本身较为复杂，maven打包为jar,再将jar以java包装为镜像)

gitlab-runner                 gitlab-runner-java-0                                    1/1     Running             0          47d
gitlab-runner                 gitlab-runner-node-0                                    1/1     Running             0          47d
gitlab-runner                 gitlab-runner-python-0                                  1/1     Running             0          47d
gitlab-runner                 gitlab-runner-golang-0                                  1/1     Running             0          47d

进入gitlab后台，获取项目的token

Screen Shot 2021-01-11 at 3.49.31 PM

进入容器以token 注册gitlab-runner

---
root@gitlab-runner-java-0:/#    gitlab-runner register -n \
>    --url https://[abc.gitlab.com] \
>    --registration-token [your_project_token] \
>    --tag-list java,prod,preview,test \
>    --executor docker \
>    --description "java" \
>    --cache-dir /data1/pv/gitlab-runner-cache/gitlab-runner-java/cache-dir \
>    --docker-cache-dir /data1/pv/gitlab-runner-cache/gitlab-runner-java/docker-cache-dir \
>    --docker-image "docker:18.06.3" \
>    --docker-pull-policy "if-not-present" \
>    --docker-volumes /var/run/docker.sock:/var/run/docker.sock \
>    --docker-volumes /root/.docker/config.json:/root/.docker/config.json \
>    --docker-volumes /data1/pv/gitlab-runner-cache/gitlab-runner-java:/cache \
>    --docker-volumes /data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository
Runtime platform                                    arch=amd64 os=linux pid=94 revision=05161b14 version=12.4.1
Running in system-mode.

Registering runner... succeeded                     runner=joTBdSTv

重点注意--docker-volumes的几项参数，因为是did的执行方式，外部目录尤其是(/root/.m2/repository)需要在maven容器持久化，k8s yaml mount的路径只是查看需要，gitlab-runner注册时的docker-volumes才是真正在不同任务间共享数据的必须

例如--docker-volumes /root/.docker/config.json:/root/.docker/config.json 宿主机docker login的授权信息，可以共享使用

​ --docker-volumes /data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository 对maven容器，不同项目共同挂载宿主机目录，减少jar包的重复下载

注册完后的内容如

root@gitlab-runner-java-0:/# cat /etc/gitlab-runner/config.toml
concurrent = 8
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "java"
  url = "https://[abc.123.com]"
  token = "[your token]"
  executor = "docker"
  cache_dir = "/data1/pv/gitlab-runner-cache/gitlab-runner-java/cache-dir"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:18.06.3"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/root/.docker/config.json:/root/.docker/config.json", "/data1/pv/gitlab-runner-cache/gitlab-runner-java:/cache", "/data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository"]
    cache_dir = "/data1/pv/gitlab-runner-cache/gitlab-runner-java/docker-cache-dir"
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

重点的项目配置文件，具体项目结构见https://github.com/cclient/hanlp-remote-dict

.
├── Dockerfile --打包docker镜像配置
├── deploy --镜像打包成功，更新上线至k8s配置
│   ├── deployment_preview.yaml
│   ├── deployment_prod.yaml
│   └── deployment_test.yaml
├── .gitlab-ci.yml --gitlab-runner ci/cd配置
├── pom.xml --java项目依赖配置

.gitlab-ci.yml

variables:
  DOCKER_API_VERSION: '1.38'
  DOCKER_HUB: "[docker hub/registry]"
  DOCKER_USERNAME: "[your docker user]"
  DOCKER_PASSWORD: "[your docker passwd]"
  PROJECT_NAME: $CI_PROJECT_NAME
  PROJECT_VERSION: $CI_COMMIT_TAG
  K8S_API_URL: "https://[k8s api host]:6443"
  K8S_ACCESS_TOKEN: "[your k8s token]"
  K8S_DEPLOY_BASE_NAME: "hanlp-remote-dict"
  CONTAINER_IMAGE: $DOCKER_HUB/$CI_PROJECT_NAME

image: docker:18.06.3
stages:
  - prepare-env
  - maven-package
  - docker-build
  - k8s-deploy

### 根据git branch/tag匹配 cicd上线环境，添加相关环境变量
prepare-env-develop:
  stage: prepare-env
  tags:
    - java
  only:
    - develop
  before_script:
    - mkdir -p ./sartifacts/
  script:
    - echo BUILD_ENV="test" >> ./sartifacts/version
    - echo BUILD_IMAGE_VERSION="test" >> ./sartifacts/version
  artifacts:
    name:  "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - ./sartifacts/*
    expire_in: 2 day

prepare-env-preview:
  stage: prepare-env
  tags:
    - java
  only:
    - /^release\/.*$/
    - master
  before_script:
    - echo docker login hub.intra.github.com -u "'$DOCKER_USERNAME'" -p $DOCKER_PASSWORD
    - mkdir -p ./sartifacts/
  script:
    - echo BUILD_ENV="preview" >> ./sartifacts/version
    - echo BUILD_IMAGE_VERSION="preview" >> ./sartifacts/version
  artifacts:
    name:  "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - ./sartifacts/*
    expire_in: 2 day

prepare-env-production:
  stage: prepare-env
  tags:
    - java
  only:
    - tags
  before_script:
    - mkdir -p ./sartifacts/
  script:
    - echo BUILD_ENV="prod" >> ./sartifacts/version
    - echo BUILD_IMAGE_VERSION=$CI_COMMIT_TAG >> ./sartifacts/version
  artifacts:
    name:  "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - ./sartifacts/*
    expire_in: 2 day

### 以下为通用部分

#### mvn 打包java项目，生成jar包，并将jar包添加至./sartifacts/ 传至下一级
maven-package:
  image: maven:3.6.3-adoptopenjdk-8
  stage: maven-package
  tags:
    - java
  only:
    - tags
    - /^release\/.*$/
    - master
    - develop
  before_script:
    # source加载./sartifacts/version的环境变量
    - source ./sartifacts/version
    # echo只是调试用，打印gitlab-runner结合git的变量
    - echo 'CI_CONFIG_PATH' $CI_CONFIG_PATH
    - echo 'CI_COMMIT_REF_NAME' $CI_COMMIT_REF_NAME
    - echo 'CI_COMMIT_BRANCH' $CI_COMMIT_BRANCH
    - echo 'CI_COMMIT_TAG' $CI_COMMIT_TAG
    - echo 'CI_COMMIT_TITLE' $CI_COMMIT_TITLE
    - echo 'CI_ENVIRONMENT_NAME' $CI_ENVIRONMENT_NAME
    - echo 'CONTAINER_IMAGE' $CONTAINER_IMAGE
    - echo 'ENV' $ENV
    - echo 'MAVEN_OPTS' "$MAVEN_OPTS"
    - echo 'BUILD_ENV' "$BUILD_ENV"
    - echo 'BUILD_IMAGE_VERSION' "$BUILD_IMAGE_VERSION"
  script:
    - mvn clean package -Dmaven.test.skip=true -Dmaven.repo.local=/root/.m2/repository
    - mv target ./sartifacts/
  artifacts:
    name:  "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - ./sartifacts/*
    expire_in: 2 day

#### 	获取上一级的sartifacts，提取target目录，打包image并上传
docker-build:
  image: docker:18.06.3
  stage: docker-build
  tags:
    - java
  only:
    - tags
    - /^release\/.*$/
    - master
    - develop
  before_script:
    - pwd
    - ls -alh
    - source ./sartifacts/version
    - mv ./sartifacts/target ./target
  script:
    - docker build -t $CONTAINER_IMAGE:$BUILD_IMAGE_VERSION ./
#    - docker login hub.intra.github.com -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker push $CONTAINER_IMAGE:$BUILD_IMAGE_VERSION

#### 执行k8s yaml 上线pod/deploy/sts等服务
k8s-deploy:
  image: telefonica/kubectl:1.15.10
  stage: k8s-deploy
  tags:
    - java
  only:
    - tags
    - /^release\/.*$/
    - master
    - develop
  before_script:
    - source ./sartifacts/version
    - echo K8S_DEPLOY_BASE_NAME-BUILD_ENV "$K8S_DEPLOY_BASE_NAME-$BUILD_ENV"
    - echo CONTAINER_IMAGE "$CONTAINER_IMAGE"
  script:
    - kubectl --server="${K8S_API_URL}" --token="$K8S_ACCESS_TOKEN" --insecure-skip-tls-verify apply -f ./deploy/deployment_$BUILD_ENV.yaml -n default
    - kubectl --server="${K8S_API_URL}" --token="$K8S_ACCESS_TOKEN" --insecure-skip-tls-verify set image deployment/$K8S_DEPLOY_BASE_NAME-$BUILD_ENV $K8S_DEPLOY_BASE_NAME-$BUILD_ENV=$CONTAINER_IMAGE:$BUILD_IMAGE_VERSION -n default --record

基本流程为

  • 1 第一阶段 prepare-env 根据gitlab 提交和分枝信息，构造version文件(保存env),供后续阶段使用，目前实现了三项prepare-env-develop，prepare-env-preview，prepare-env-production，内容一样是写入version,为了保证这里不够精简，看了下gitlab-runner的rule，应该可以有更优雅的方案，但个人没有精力深入，有知道更好办法的，烦请告知。
  • 2 第二阶段 maven-package did以maven:3.6.3-adoptopenjdk-8为底包打包项目，生成target目录(下有jar包)，并将targer整个打包为sartifacts传给下一阶段，大包的话，只用jar即可
  • 3 第三阶段 docker-build dit以docker:18.06.3为底包，自动下载解压sartifacts，拷备出target至项目目标地址，根据Dockerfile(对该示例而言底包为openjdk:8u162-jre-slim-stretch) 生成image 并上传,docker login hub.intra.github.com -u $DOCKER_USERNAME -p $DOCKER_PASSWORD 这一步如果挂载了宿主机的docker 授权则不必要
  • 4 第四阶段 k8s-deploy did 以telefonica/kubectl:1.15.10为底包,因需选择版本，个人用该镜像从k8s v1.15.0用到v1.18.5，目前使用正常，通过api 提交相应deploy.yaml至k8s api，这一阶段执行两部，kubectl apply -f deploy.yaml 提交上线，再kubectl set image 更新deploy内服务image的真实版本

docker file

FROM openjdk:8u162-jre-slim-stretch
WORKDIR /home/spring
ADD target/*.jar /home/spring/app.jar

deploy.yaml 以测试deploy为例

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hanlp-remote-dict-test
  name: hanlp-remote-dict-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hanlp-remote-dict-test
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: hanlp-remote-dict-test
    spec:
      containers:
        - image: cclient/arm64v8-kafka:test
          command:
            - java
          args:
            - -Xms1g
            - -Xmx2g
            - -XX:+UseG1GC
            - -XX:G1ReservePercent=10
            - -XX:+UseStringDeduplication
            - -jar
            - app.jar
          imagePullPolicy: Always
          name: hanlp-remote-dict-test
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: hanlp-remote-dict-test
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: hanlp-remote-dict-test
  sessionAffinity: None
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hanlp-remote-dict-test
  namespace: default
spec:
  rules:
    - host: hanlp-remote-dict-test.github.com
      http:
        paths:
          - path: /
            backend:
              serviceName: hanlp-remote-dict-test
              servicePort: 8080

执行成功样例

Screen Shot 2021-01-11 at 5.58.36 PM

[session_server].listen_address not defined, session endpoints disabled  builds=0
Checking for jobs... received                       job=196962 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
Job succeeded                                       duration=21.427262629s job=196962 project=11190 runner=sdWbANrN
Checking for jobs... received                       job=196963 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
Job succeeded                                       duration=2m3.362519176s job=196963 project=11190 runner=sdWbANrN
Checking for jobs... received                       job=196964 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN
Job succeeded                                       duration=4m8.502822318s job=196964 project=11190 runner=sdWbANrN
Checking for jobs... received                       job=196965 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN

其他

1 目前spring boot 构造为一个大包，docker image上传下载io开销较大，可调整为先下载准备了依赖，再打小包

2 个人没有使用https://github.com/fabric8io/docker-maven-plugin等mvn 组件，主要原因是要支持多语言，方案不同，但共用通用的ci/cd抽像，对其他语言而言，可能不存在相同定位的包管理插件，docker cmd 更通用，对环境的依赖也小

3 test,preview,prod 不同上线分枝可以提交不同的 application.properties/application.yaml配置

限于篇幅太长，最后test,preview,prod对应git 分枝管理，以后再弹独写一篇

End

Screen Shot 2021-01-15 at 1.46.47 PM

该方案将maven打包jar文件，以jar构建docker镜像分成了两个stage

优点时可以在gitlab-runner内下载jar包,在没有自建maven中心的条件下，本地缓存复用maven jar包(跨项目)，缺点是jar文件的传输有额外的开销，两步之前通过网络io,gitlab网络io较慢会较为耗时

可以利用docker原生的 multi-stage builds功能

在一个staget内完成打包jar文件，生成docker镜像的功能

https://docs.docker.com/develop/develop-images/multistage-build/

这个方案也有，也更简单，有时间再整理吧(缺点是，无法映射本地jar包，每次都要重要下载mvn依赖，因此需，默认国外源的速度大家都懂，因此需把国外源调整为国内源和自建mirror源)

posted @ 2021-02-28 16:59  cclient  阅读(1225)  评论(0)  编辑  收藏  举报