基于Kubernetes构建Jenkins微服务发布平台

发布流程设计

软件环境：Jenkins + Kubernetes + Gitlab + Harbor+helm

工作流程：手动/自动构建-> Jenkins 调度K8S API-＞动态生成Jenkins Slave pod -＞Slave pod 拉取Git 代码／编译／打包镜像-＞推送到镜像仓库Harbor -＞Slave 工作完成，Pod 自动销毁-＞helm部署到测试或生产Kubernetes平台。

准备基础前提环境

1、K8s（Ingress Controller，CoreDNS，PV自动供给）
2、Helm v3
3、Gitlab
4、Harbor，并启用Chart存储功能
5、MySQL（微服务数据库）
6、Eureka（注册中心）

在Kubernetes中部署Jenkins

参考：https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

安装插件: Git Parameter/Git/Pipeline/Config File Provider/kubernetes/Extended Choice Parameter

由于默认插件源在国外服务器，大多数网络无法顺利下载，需修改国内插件源地址：

cd jenkins_home/updates
sed -i 's/http:\/\/updates.jenkins-ci.org\/download/https:\/\/mirrors.tuna.tsinghua.edu.cn\/jenkins/g' default.json && \
sed -i 's/http:\/\/www.google.com/https:\/\/www.baidu.com/g' default.json

Jenkins Pipeline 及参数化构建

参考：https://jenkins.io/doc/book/pipeline/syntax/

• Jenkins Pipeline是一套插件，支持在Jenkins中实现集成和持续交付管道；
• Pipeline通过特定语法对简单到复杂的传输管道进行建模；
• 声明式：遵循与Groovy相同语法。pipeline { }
• 脚本式：支持Groovy大部分功能，也是非常表达和灵活的工具。node { }
• Jenkins Pipeline的定义被写入一个文本文件，称为Jenkinsfile。

Jenkins在Kubernetes中动态创建代理

Jenkins Master/Slave架构

在K8S中Jenkins Master/Slave架构

Kubernetes插件：Jenkins在Kubernetes集群中运行动态代理
插件介绍：https://github.com/jenkinsci/kubernetes-plugin

自定义构建Jenkins Slave镜像

参考：https://github.com/jenkinsci/docker-jnlp-slave

FROM centos:7
LABEL maintainer ops

RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \
    yum clean all && \
    rm -rf /var/cache/yum/* && \
    mkdir -p /usr/share/jenkins

COPY slave.jar /usr/share/jenkins/slave.jar
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave
COPY helm kubectl /usr/bin/

ENTRYPOINT ["jenkins-slave"]

构建jenkins-slave推送至harbor仓库

docker build . -t 192.168.0.241/library/jenkins-slave:jdk-1.8
docker push 192.168.0.241/library/jenkins-slave:jdk-1.8

 基于Kubernetes构建Jenkins CI系统

添加凭据

jenkinsfile

拉取代码 => 代码编译 => 单元测试 => 构建镜像 => Helm部署到K8S => 测试

#!/usr/bin/env groovy

// 公共
def registry = "192.168.0.241"
// 项目
def project = "microservice"
def git_url = "http://192.168.0.138:12580/root/ms.git"
def gateway_domain_name = "gateway.xxxx.com"
def portal_domain_name = "portal.xxxx.com"
// 认证
def image_pull_secret = "registry-pull-secret"
def harbor_registry_auth = "c176b2b2-7eaf-4968-a1e0-8988d6e04057"
def git_auth = "03e08854-c546-43c1-82c9-d053a75953d2"
// ConfigFileProvider ID
def k8s_auth = "6e3f7dd8-6e26-42de-91c3-5b52ff3117ec"

pipeline {
  agent {
    kubernetes {
        label "jenkins-slave"
        yaml """
kind: Pod
metadata:
  name: jenkins-slave
spec:
  containers:
  - name: jnlp
    image: "${registry}/library/jenkins-slave:jdk-1.8"
    imagePullPolicy: Always
    volumeMounts:
      - name: docker-cmd
        mountPath: /usr/bin/docker
      - name: docker-sock
        mountPath: /var/run/docker.sock
      - name: maven-cache
        mountPath: /root/.m2
  volumes:
    - name: docker-cmd
      hostPath:
        path: /usr/bin/docker
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock
    - name: maven-cache
      hostPath:
        path: /tmp/m2
"""
        }
      
      }
    parameters {
        gitParameter branch: '', branchFilter: '.*', defaultValue: 'master', description: '选择发布的分支', name: 'Branch', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH'        
        extendedChoice defaultValue: 'none', description: '选择发布的微服务', \
          multiSelectDelimiter: ',', name: 'Service', type: 'PT_CHECKBOX', \
          value: 'gateway-service:9999,portal-service:8080,product-service:8010,order-service:8020,stock-service:8030'
        choice (choices: ['ms', 'demo'], description: '部署模板', name: 'Template')
        choice (choices: ['1', '3', '5', '7'], description: '副本数', name: 'ReplicaCount')
        choice (choices: ['ms'], description: '命名空间', name: 'Namespace')
    }
    stages {
        stage('拉取代码'){
            steps {
                checkout([$class: 'GitSCM', 
                branches: [[name: "${params.Branch}"]], 
                doGenerateSubmoduleConfigurations: false, 
                extensions: [], submoduleCfg: [], 
                userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_url}"]]
                ])
            }
        }
        stage('代码编译') {
            // 编译指定服务
            steps {
                sh """
                  mvn clean package -Dmaven.test.skip=true
                """
            }
        }
        stage('构建镜像') {
          steps {
              withCredentials([usernamePassword(credentialsId: "${harbor_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
                sh """
                 docker login -u ${username} -p '${password}' ${registry}
                 for service in \$(echo ${Service} |sed 's/,/ /g'); do
                    service_name=\${service%:*}
                    image_name=${registry}/${project}/\${service_name}:${BUILD_NUMBER}
                    cd \${service_name}
                    if ls |grep biz &>/dev/null; then
                        cd \${service_name}-biz
                    fi
                    docker build -t \${image_name} .
                    docker push \${image_name}
                    cd ${WORKSPACE}
                  done
                """
                configFileProvider([configFile(fileId: "${k8s_auth}", targetLocation: "admin.kubeconfig")]){
                    sh """
                    # 添加镜像拉取认证
                    kubectl create secret docker-registry ${image_pull_secret} --docker-username=${username} --docker-password=${password} --docker-server=${registry} -n ${Namespace} --kubeconfig admin.kubeconfig |true
                    # 添加私有chart仓库
                    helm repo add  --username ${username} --password ${password} myrepo http://${registry}/chartrepo/${project}
                    """
                }
              }
          }
        }
        stage('Helm部署到K8S') {
          steps {
              sh """
              common_args="-n ${Namespace} --kubeconfig admin.kubeconfig"
              
              for service in  \$(echo ${Service} |sed 's/,/ /g'); do
                service_name=\${service%:*}
                service_port=\${service#*:}
                image=${registry}/${project}/\${service_name}
                tag=${BUILD_NUMBER}
                helm_args="\${service_name} --set image.repository=\${image} --set image.tag=\${tag} --set replicaCount=${replicaCount} --set imagePullSecrets[0].name=${image_pull_secret} --set service.targetPort=\${service_port} myrepo/${Template}"

                # 判断是否为新部署
                if helm history \${service_name} \${common_args} &>/dev/null;then
                  action=upgrade
                else
                  action=install
                fi

                # 针对服务启用ingress
                if [ \${service_name} == "gateway-service" ]; then
                  helm \${action} \${helm_args} \
                  --set ingress.enabled=true \
                  --set ingress.host=${gateway_domain_name} \
                   \${common_args}
                elif [ \${service_name} == "portal-service" ]; then
                  helm \${action} \${helm_args} \
                  --set ingress.enabled=true \
                  --set ingress.host=${portal_domain_name} \
                   \${common_args}
                else
                  helm \${action} \${helm_args} \${common_args}
                fi
              done
              # 查看Pod状态
              sleep 10
              kubectl get pods \${common_args}
              """
          }
        }
    }
}

 

Pipeline 集成Helm 发布微服务项目

小结：

❖使用Jenkins的插件
  •Git & gitParameter
  •Kubernetes
  •Pipeline
  •Kubernetes Continuous Deploy
  •Config File Provider
  •Extended Choice Parameter
❖CI/CD环境特点
  •Slave弹性伸缩
  •基于镜像隔离构建环境
  •流水线发布，易维护
❖Jenkins参数化构建可帮助你完成更复杂环境CI/CD