kubernetes下Jenkins分布式部署：jenkinfiles--maven编译+镜像推送+sonar代码扫描+部署+企业微信通知

k8s下Jenkins分部署部署：jenkinfiles--maven编译+镜像推送+sonar代码扫描+部署+企业微信通知

准备好k8s集群、安装好Jenkins、准备gitlab的ssh密钥、准备k8s的config、安装sonar

k8s集群

• k8s集群说明
  k8s的node01节点打标签，build，用于运行Jenkins的pod。
  
  本地创建/data/jenkins 和 /data/m2 ,用于给Jenkins的pod做数据持久化，注意给目录添加Jenkins属主。
  

• k8s的config文件：
  在k8s服务器master01节点上下载config文件

cp /usr/local/kubernetes/cfg/admin.kubeconfig ./config
sz config

这个config等会需要上传到jenkins上。

Jenkins部署

centos7服务器上使用yum安装Jenkins

java环境准备

版本：jdk-11.0.14

Jenkins安装

# 下载 Jenkins repo 文件
wget --no-check-certificate -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

# 导入秘钥
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

# 安装 jdk，可选 Oracle jdk 或者 Openjdk
# yum install epel-release java-11-openjdk-devel

# 查看 Jenkins 可选版本
yum makecache fast
yum list jenkins --showduplicates 
# 安装 Jenkins
yum install jenkins fontconfig -y
修改 /usr/lib/systemd/system/jenkins.service 文件，取消并修改以下配置注释
Environment="JENKINS_LOG=/var/log/jenkins/jenkins.log"
Environment="JAVA_HOME=/usr/local/java/jdk-11.0.13" # 修改具体的 JDK 安装目录
# 配置开机启动
systemctl daemon-reload && chkconfig jenkins on

# 启动 Jenkinsls 
systemctl start jenkins.service 

# 查看 jenkins 服务状态
systemctl status jenkins.service
配置 Jenkins，通过 yum 安装的 jenkins，相关文件默认所在位置

家目录: /var/lib/jenkins/
配置文件: /etc/sysconfig/jenkins
日志文件: /var/log/jenkins/jenkins.log
初始密码文件: /var/lib/jenkins/secrets/initialAdminPassword

配置修改：
vi /usr/lib/systemd/system/jenkins.service

添加Jenkins账户、给/data/jenkins授权：chown jenkins:jenkins /data/jenkins -R

Jenkins插件和配置

系统配置

• 插件管理

Ant Plugin
Autofavorite for Blue Ocean
Bitbucket Pipeline for Blue Ocean
Blue Ocean Pipeline Editor
Build Timeout
build user vars plugin
Command Agent Launcher Plugin
Config API for Blue Ocean
Display URL for Blue Ocean
Email Extension Plugin
Events API for Blue Ocean
Git Changelog
Git Parameter Plug-In
Git Pipeline for Blue Ocean
Git server Plugin
GitHub Pipeline for Blue Ocean
GitLab
GitLab Authentication plugin
GitLab Branch Source
Gradle
Groovy
i18n for Blue Ocean
Image Tag Parameter Plugin
Kubernetes
Kubernetes CLI Plugin
LDAP Plugin
Localization: Chinese (Simplified)
Matrix Authorization Strategy Plugin
Oracle Java SE Development Kit Installer
PAM Authentication plugin
Personalization for Blue Ocean
Pipeline
Pipeline: GitHub Groovy Libraries
Pipeline: Stage View
Qy Wechat Notification Plugin
Role-based Authorization Strategy
SonarQube Scanner for Jenkins
SSH Agent Plugin
SSH Build Agents
Timestamper
Workspace Cleanup Plugin

• 系统配置
  主目录：/data/jenkins
  Jenkins Location --> Jenkins URL ： http://10.10.246.29:8080/
  SonarQube servers :
  
• 全局工具配置
  无需配置
• 节点管理
  configureClouds ：
  

安全

• 全局安全配置
  
  
  
• Manage Credentials
  
  将master01节点上下载config文件上传到凭据中：
  
  将sonar的密钥填入：
  
  sonar的密钥token生成方式：进入我的账号-->安全，填写新令牌名称，点击生成，复制生成的token
  
  将gitlab的ssh密钥填入：
  
  
  将镜像仓库的账户密码填入：
  

Manage and Assign Roles

• Manage Roles
  
  

• Assign Roles
  
  

其他配置默认即可

开发环境发布

Jenkinsfile说明

• 后端java工程

pipeline {
  agent {
    kubernetes {
      cloud 'kubernetes-dev'
      slaveConnectTimeout 1200
      workspaceVolume hostPathWorkspaceVolume(hostPath: "/data/jenkins/workspace", readOnly: false)
      yaml '''
apiVersion: v1
kind: Pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/edge
            operator: DoesNotExist
          - key: build
            operator: In
            values:
            - "true"
  tolerations:
  - key: "groups"
    operator: "Equal"
    effect: "NoExecute"
    value: "haian"
  containers:
  - name: jnlp
    image: \'10.10.246.29/citools/jnlp:latest-jdk11\'
    imagePullPolicy: IfNotPresent
    args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    - mountPath: "/home/jenkins/.ssh"
      name: "sshd"
  - name: "build"
    image: "10.10.246.29/citools/maven:3.8.6-openjdk-8"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
    - mountPath: "/root/.m2/"
      name: "cachedir"
      readOnly: false
  - name: "sonar-scanner"
    image: "10.10.246.29/citools/sonar-scanner-cli:4.7"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    - name: "SONAR_HOST_URL"
      value: "http://10.10.246.29:19000/"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
  - name: "kubectl"
    image: "10.10.246.29/citools/kubectl:self-1.17"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  - name: "docker"
    image: "10.10.246.29/citools/docker:19.03.9-git"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "dockersock"
      readOnly: false
    - mountPath: /var/jenkins/workspace
      name: "jenkins-workspace"
      readOnly: false
  - name: "alpine-curl"
    image: "10.10.246.29/citools/alpine-curl:latest"
    imagePullPolicy: "IfNotPresent"
    command:
    - "/bin/sh"
    - "-c"
    - "sleep 1200"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    volumeMounts:
    - mountPath: /var/jenkins/workspace
      name: "jenkins-workspace"
      readOnly: false
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: true
  restartPolicy: "Never"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "dockersock"
  - hostPath:
      path: "/usr/share/zoneinfo/Asia/Shanghai"
    name: "localtime"
  - name: "cachedir" 
    hostPath:
      path: "/data/m2"
  - name: "jenkins-workspace"
    hostPath:
      path: "/data/jenkins/workspace"
  - name: "sshd"
    hostPath: 
      path: "/data/jenkins/sshd-secrets"
'''
    }
  }

  environment {
    TAG = ''
    COMMIT_ID = ''
    HARBOR_ADDRESS = '10.10.246.29'
    HARBOR_USER = credentials('HARBOR_ACCOUNT')
    REGISTRY_DIR = 'haian'
    IMAGE_NAME = 'vchain-afterloan'
    NAMESPACE = 'haian-dev'
    MY_KUBECONFIG = credentials('kubernetes-haian-dev')
    REPO = 'ssh://git@gitlab.vonechain.com:22022/haianSCF/vchain-scf.git'
    REPO_HTTP = 'http://gitlab.vonechain.com:9002/haianSCF/vchain-scf.git'
    PROJECT_DIR = 'vchain-afterloan/vchain-afterloan-biz'
    GIT_AUTH = 'gitlab-key'
    LANGUAGE = 'java'
  }
  
  parameters {
    gitParameter(
      branch: '',
      branchFilter: 'origin/(.*)', 
      defaultValue: 'dev', 
      description: 'Branch for build and deploy', 
      name: 'BRANCH', 
      quickFilterEnabled: false, 
      selectedValue: 'NONE', 
      sortMode: 'NONE', 
      tagFilter: '*', 
      type: 'PT_BRANCH'
    )
    imageTag(
      name: "IMAGE_TAG", 
      description: '需要提测的镜像版本，构建发版开发环境时请忽略该参数',
      image: "haian/vchain-afterloan", 
      filter: '.*', 
      defaultTag: '',
      registry: 'http://10.10.246.29', 
      credentialId: 'HARBOR_ACCOUNT', 
      tagOrder: 'DSC_VERSION'
    )
    choice(
      name: 'ACTION',
      description: '执行动作: \nbuild: 只构建镜像，不发版\ndeploy: 构建镜像并发布更新到开发环境\nsummit_the_test: 提交测试申请',
      choices: ['deploy','summit_the_test','build']
    )
    booleanParam(
      name:'SCANNER',
      defaultValue: false,
      description: '是否执行代码扫描'
    )

    text(
      defaultValue: '''此次提测需要注意以下事项:''', 
      description: '提测信息说明', 
      name: 'COMMENT'
    )
  }

  stages {
    stage('Pulling Code') {
      when {
        expression { params.ACTION ==~ /(build|deploy)/ }
      }
      parallel {
        stage('Pulling Code by Jenkins') {
          when {
            expression {
              env.gitlabBranch == null
            }
          }
          steps {
            git(
              url: "${env.REPO}", 
              changelog: true, 
              poll: true, 
              branch: "${BRANCH}", 
              credentialsId: "${env.GIT_AUTH}"
            )
            script {
              COMMIT_ID = sh(
                returnStdout: true, 
                script: "git log -n 1 --pretty=format:'%h'"
              ).trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }
          }
        }

        stage('Pulling Code by trigger') {
          when {
            expression {
              env.gitlabBranch != null
            }
          }
          steps {
            git(
              url: "${env.REPO}", 
              branch: env.gitlabBranch, 
              changelog: true, 
              poll: true, 
              credentialsId: "${env.GIT_AUTH}"
            )
            script {
              COMMIT_ID = sh(
                returnStdout: true,
                script: "git log -n 1 --pretty=format:'%h'"
              ).trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }
          }
        }

      }
    }

    stage('Building') {
      when {
        expression { params.ACTION ==~ /(build|deploy)/ }
      }
      steps {
        container(name: 'build') {
          sh '''
            cd ${IMAGE_NAME}
            curl -sSL repo.maven.apache.org 
            mvn clean package -U -B -DskipTests=true
          '''
        }
      }
    }


    stage('SonarQube analysis') {
      environment {
        SONAR_TOKEN = credentials('sonar_token')
      }
      when {
        expression { return params.SCANNER }
      }

      steps {
        container("sonar-scanner") {
          sh '''
            sonar-scanner \
              -Dsonar.login=${SONAR_TOKEN} \
              -Dsonar.projectKey=${IMAGE_NAME} \
              -Dsonar.projectName=${IMAGE_NAME} \
              -Dsonar.sourceEncoding=UTF-8 \
              -Dsonar.language=${LANGUAGE} \
              -Dsonar.sources=${PROJECT_DIR} \
              -Dsonar.java.binaries=.
          '''
        }
      }
    }

    stage('Docker build for creating image') {
      when {
        expression { params.ACTION ==~ /(build|deploy)/ }
      }
      steps {
        container(name: 'docker') {
          sh """
            cd ${PROJECT_DIR}
            echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
            docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
            docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
            docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
          """
        }
      }
      post {
        always {
          container(name: 'docker') {
            sh """
              echo "清理编译文件"
              rm -rf ./target
            """
          }
        }
      }
    }

    stage('Deploying to K8s') {
      when {
        expression { params.ACTION == 'deploy' }
      }
      steps {
        container(name: 'kubectl') {
          sh """
            /usr/local/bin/kubectl --kubeconfig ${MY_KUBECONFIG} \
              set image deploy -l \
              app=${IMAGE_NAME} \
              ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} \
              -n $NAMESPACE --record
            /usr/local/bin/kubectl --kubeconfig=${MY_KUBECONFIG} \
              get pod -l app=${IMAGE_NAME} \
              -n ${NAMESPACE} -w
          """
        }
      }
    }

    stage('提交到测试') {
      when {
        expression { params.ACTION == 'summit_the_test' }
      }

      steps {
        container(name: 'alpine-curl') {
          wrap([$class: 'BuildUser']) {
            script {
              def MSG = '''https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=9999999999999999-5d160ecef9a9'''
              BUILD_USER = "${env.BUILD_USER}"
              IMAGE_TAG = "${params.IMAGE_TAG}"
              BUILD_TIMESTAMP = "${env.BUILD_TIMESTAMP}"
              sh """
                echo "发送提测消息到企业微信."
                curl '${MSG}' -H 'Content-Type: application/json' -d '
                {
                  "msgtype": "markdown",
                  "markdown": {
                    "content": "`项目 <font color=\'info\'>${IMAGE_NAME}</font> 已经提测，请相关同事注意。`\n> 
                               > 提测项目名称: <font color=\'info\'>${IMAGE_NAME}</font>
                               > 提测代码仓库: [${IMAGE_NAME}](${env.REPO_HTTP})
                               > 提测代码分支: <font color=\'warning\'>${BRANCH}</font>
                               > 提测镜像版本: ${IMAGE_TAG}
                               > 提测镜像标签: <font color=\'warning\'>${IMAGE_TAG_TAG}</font>
                               > 提测镜像名称: <font color=\'warning\'>${IMAGE_TAG_IMAGE}</font>
                               > 提测申请人: <font color=\'info\'>${BUILD_USER}</font>
                               > 提测申请时间: <font color=\'comment\'>${BUILD_TIMESTAMP}</font>
                               > 测试环境部署: 确保nacos配置已经同步，请点击：[发布测试环境](http://10.10.246.29:8080/)
                               > 提测注释: ${params.COMMENT}"
                  }
                }'
              """
            }
          }
        }
      }
    }
  }
}

• 前端node工程

pipeline {
  agent {
    kubernetes {
      cloud 'kubernetes-dev'
      slaveConnectTimeout 1200
      workspaceVolume hostPathWorkspaceVolume(hostPath: "/data/jenkins/workspace", readOnly: false)
      yaml '''
apiVersion: v1
kind: Pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/edge
            operator: DoesNotExist
          - key: build
            operator: In
            values:
            - "true"
  tolerations:
  - key: "groups"
    operator: "Equal"
    effect: "NoExecute"
    value: "haian"
  containers:
  - name: jnlp
    image: \'10.10.246.29/citools/jnlp:latest-jdk11\'
    imagePullPolicy: IfNotPresent
    args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    - mountPath: "/home/jenkins/.ssh"
      name: "sshd"
  - name: "build"
    image: "10.10.246.29/citools/node:14.17.1"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"

  - name: "sonar-scanner"
    image: "10.10.246.29/citools/sonar-scanner-cli:4.7"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    - name: "SONAR_HOST_URL"
      value: "http://10.10.246.29:19000/"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      
  - name: "kubectl"
    image: "10.10.246.29/citools/kubectl:self-1.17"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  - name: "docker"
    image: "10.10.246.29/citools/docker:19.03.9-git"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "dockersock"
      readOnly: false
    - mountPath: /var/jenkins/workspace
      name: "jenkins-workspace"
      readOnly: false
  - name: "alpine-curl"
    image: "10.10.246.29/citools/alpine-curl:latest"
    imagePullPolicy: "IfNotPresent"
    command:
    - "/bin/sh"
    - "-c"
    - "sleep 1200"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    volumeMounts:
    - mountPath: /var/jenkins/workspace
      name: "jenkins-workspace"
      readOnly: false
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: true
  restartPolicy: "Never"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "dockersock"
  - hostPath:
      path: "/usr/share/zoneinfo/Asia/Shanghai"
    name: "localtime"
  - name: "cachedir" 
    hostPath:
      path: "/data/m2"
  - name: "jenkins-workspace"
    hostPath:
      path: "/data/jenkins/workspace"
  - name: "sshd"
    hostPath: 
      path: "/data/jenkins/sshd-secrets"
'''
    }
  }

  environment {
    TAG = ''
    COMMIT_ID = ''
    HARBOR_ADDRESS = '10.10.246.29'
    HARBOR_USER = credentials('HARBOR_ACCOUNT')
    REGISTRY_DIR = 'haian'
    IMAGE_NAME = 'vchain-scf-admin-front'
    NAMESPACE = 'haian-dev'
    MY_KUBECONFIG = credentials('kubernetes-haian-dev')
    REPO = 'ssh://git@gitlab.vonechain.com:22022/haianSCF/vchain-scf-admin-front.git'
    REPO_HTTP = 'http://gitlab.vonechain.com:9002/haianSCF/vchain-scf-admin-front.git'
    GIT_AUTH = 'gitlab-key'
    SOURCES_DIR = '.'
    LANGUAGE = 'javascript'

  }
  
  parameters {
    gitParameter(
      branch: '',
      branchFilter: 'origin/(.*)', 
      defaultValue: 'dev', 
      description: 'Branch for build and deploy', 
      name: 'BRANCH', 
      quickFilterEnabled: false, 
      selectedValue: 'NONE', 
      sortMode: 'NONE', 
      tagFilter: '*', 
      type: 'PT_BRANCH'
    )
    imageTag(
      name: "IMAGE_TAG", 
      description: '需要提测的镜像版本，构建发版开发环境时请忽略该参数',
      image: "haian/vchain-scf-admin-front", 
      filter: '.*', 
      defaultTag: '',
      registry: 'http://10.10.246.29', 
      credentialId: 'HARBOR_ACCOUNT', 
      tagOrder: 'DSC_VERSION'
    )
    choice(
      name: 'ACTION',
      description: '执行动作: \nbuild: 只构建镜像，不发版\ndeploy: 构建镜像并发布更新到开发环境\nsummit_the_test: 提交测试申请',
      choices: ['deploy', 'build', 'summit_the_test']
    )
    booleanParam(
      name:'INSTALL',
      defaultValue: false, 
      description: '是否安装依赖'
    )

    booleanParam(
      name:'SCANNER',
      defaultValue: false,
      description: '是否执行代码扫描'
    )

    text(
      defaultValue: '''此次提测需要注意以下事项:''', 
      description: '提测信息说明', 
      name: 'COMMENT'
    )
  }

  stages {
    stage('Pulling Code') {
      when {
        expression { params.ACTION ==~ /(build|deploy)/ }
      }
      parallel {
        stage('Pulling Code by Jenkins') {
          when {
            expression {
              env.gitlabBranch == null
            }
          }
          steps {
            git(
              url: "${env.REPO}", 
              changelog: true, 
              poll: true, 
              branch: "${BRANCH}", 
              credentialsId: "${env.GIT_AUTH}"
            )
            script {
              COMMIT_ID = sh(
                returnStdout: true, 
                script: "git log -n 1 --pretty=format:'%h'"
              ).trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }
          }
        }

        stage('Pulling Code by trigger') {
          when {
            expression {
              env.gitlabBranch != null
            }
          }
          steps {
            git(
              url: "${env.REPO}", 
              branch: env.gitlabBranch, 
              changelog: true, 
              poll: true, 
              credentialsId: "${env.GIT_AUTH}"
            )
            script {
              COMMIT_ID = sh(
                returnStdout: true,
                script: "git log -n 1 --pretty=format:'%h'"
              ).trim()
              TAG = BUILD_TAG + '-' + COMMIT_ID
              println "Current branch is ${env.gitlabBranch}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"
            }
          }
        }

      }
    }

    stage('Building') {
      when {
        expression { params.ACTION ==~ /(build|deploy)/ }
      }
      steps {
        container(name: 'build') {
          script {
            if (params.INSTALL) {
              sh """
                npm install chromedriver --chromedriver_cdnurl=http://cdn.npm.taobao.org/dist/chromedriver
                npm config set registry https://registry.npm.taobao.org
                npm install
              """
            }
            sh """
              npm run build
            """
          }
        }
      }
    }

    stage('SonarQube analysis') {
      environment {
        SONAR_TOKEN = credentials('sonar_token')
      }
      when {
        expression { return params.SCANNER }
      }

      steps {
        container("sonar-scanner") {
          sh '''
            sonar-scanner \
              -Dsonar.login=${SONAR_TOKEN} \
              -Dsonar.projectKey=${IMAGE_NAME} \
              -Dsonar.projectName=${IMAGE_NAME} \
              -Dsonar.sourceEncoding=UTF-8 \
              -Dsonar.language=${LANGUAGE} \
              -Dsonar.sources=${SOURCES_DIR}
          '''
        }
      }
    }    
    
    stage('Docker build for creating image') {
      when {
        expression { params.ACTION ==~ /(build|deploy)/ }
      }
      steps {
        container(name: 'docker') {
          sh """
            echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}
            docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .
            docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
            docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}
          """
        }
      }
      post {
        always {
          container(name: 'docker') {
            sh """
              echo "清理编译文件"
              rm -rf ./dist
            """
        }
        }
      }
    }

    stage('Deploying to K8s') {
      when {
        expression { params.ACTION == 'deploy' }
      }
      steps {
        container(name: 'kubectl') {
          sh """
            /usr/local/bin/kubectl --kubeconfig ${MY_KUBECONFIG} \
              set image deploy -l \
              app=${IMAGE_NAME} \
              ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} \
              -n $NAMESPACE --record
            /usr/local/bin/kubectl --kubeconfig=${MY_KUBECONFIG} \
              get pod -l app=${IMAGE_NAME} \
              -n ${NAMESPACE} -w
          """
        }
      }
    }

    stage('提交到测试') {
      when {
        expression { params.ACTION == 'summit_the_test' }
      }

      steps {
        container(name: 'alpine-curl') {
          wrap([$class: 'BuildUser']) {
            script {
              BUILD_USER = "${env.BUILD_USER}"
              IMAGE_TAG = "${params.IMAGE_TAG}"
              BUILD_TIMESTAMP = "${env.BUILD_TIMESTAMP}"
              sh """
                echo "发送提测消息到企业微信."
                curl 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=9999999999995d160ecef9a9' -H 'Content-Type: application/json' -d '
                {
                  "msgtype": "markdown",
                  "markdown": {
                    "content": "`项目 <font color=\'info\'>${IMAGE_NAME}</font> 已经提测，请相关同事注意（该信息为调试，请忽略！！）。`\n> 
                               > 提测项目名称: <font color=\'info\'>${IMAGE_NAME}</font>
                               > 提测代码仓库: [${IMAGE_NAME}](${env.REPO_HTTP})
                               > 提测代码分支: <font color=\'warning\'>${BRANCH}</font>
                               > 提测镜像版本: ${IMAGE_TAG}
                               > 提测镜像标签: <font color=\'warning\'>${IMAGE_TAG_TAG}</font>
                               > 提测镜像名称: <font color=\'warning\'>${IMAGE_TAG_IMAGE}</font>
                               > 提测申请人: <font color=\'info\'>${BUILD_USER}</font>
                               > 提测申请时间: <font color=\'comment\'>${BUILD_TIMESTAMP}</font>
                               > 测试环境部署: 确保nacos配置已经同步，请点击：[发布测试环境](http://10.1.30.91:8080/)
                               > 提测注释: ${params.COMMENT}"
                  }
                }'
              """
            }
          }
        }
      }
    }
  }
}

流水线配置

Jenkinsfile的路径在工程下，分支是dev:

部署开发效果

第一步先选择build：

将build出来的镜像获取后，编辑yaml清单，启动pod。

只有pod启动后，才能选择deploy进行发布更新，因为Jenkinsfile中发布更新是通过命令方式

 /usr/local/bin/kubectl --kubeconfig ${MY_KUBECONFIG} \
              set image deploy -l \
              app=${IMAGE_NAME} \
              ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} \
              -n $NAMESPACE --record
            /usr/local/bin/kubectl --kubeconfig=${MY_KUBECONFIG} \
              get pod -l app=${IMAGE_NAME} \
              -n ${NAMESPACE} -w

提测

提测通知效果：

代码扫描

在sonar上添加工程名称：

工程名称与Jenkinsfile中的IMAGE_NAME一致，工程key也一致：


代码扫描结果：

测试环境发布

添加流水线

流水线内容：

pipeline {
agent {
    kubernetes {
      cloud 'kubernetes-dev'
      slaveConnectTimeout 1200
      workspaceVolume hostPathWorkspaceVolume(hostPath: "/data/jenkins/workspace", readOnly: false)
      yaml '''
apiVersion: v1
kind: Pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/edge
            operator: DoesNotExist
          - key: build
            operator: In
            values:
            - "true"
  tolerations:
  - key: "groups"
    operator: "Equal"
    effect: "NoExecute"
    value: "haian"
  containers:
  - name: jnlp
    image: \'10.10.246.29/citools/jnlp:latest-jdk11\'
    imagePullPolicy: IfNotPresent
    args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
    - mountPath: "/home/jenkins/.ssh"
      name: "sshd"

  - name: "kubectl"
    image: "10.10.246.29/citools/kubectl:self-1.17"
    imagePullPolicy: "IfNotPresent"
    command:
    - "cat"
    env:
    - name: "LANGUAGE"
      value: "en_US:en"
    - name: "LC_ALL"
      value: "en_US.UTF-8"
    - name: "LANG"
      value: "en_US.UTF-8"
    tty: true
    volumeMounts:
    - mountPath: "/etc/localtime"
      name: "localtime"
      readOnly: false
  restartPolicy: "Never"
  volumes:
  - hostPath:
      path: "/usr/share/zoneinfo/Asia/Shanghai"
    name: "localtime"
  - name: "sshd"
    hostPath: 
      path: "/data/jenkins/sshd-secrets"
'''
    }
  }

  environment {
    TAG = ""
    HARBOR_ADDRESS = "10.10.246.29"
    NAMESPACE = "haian-sit"
    IMAGE_NAME = "maorong-auth"
    MY_KUBECONFIG = credentials('kubernetes-haian-dev')
  }
  
  stages {
    stage('Deploy') {
      steps {
        container(name: 'kubectl'){
          sh """
            # 该变量即为前台选择的镜像
            echo ${IMAGE_TAG} 
            kubectl --kubeconfig=${MY_KUBECONFIG} \
              set image deployment -l app=${IMAGE_NAME} \
              ${IMAGE_NAME}=${HARBOR_ADDRESS}/${IMAGE_TAG} \
              -n ${NAMESPACE} --record
            kubectl --kubeconfig=${MY_KUBECONFIG} \
              get pod -l app=${IMAGE_NAME} \
              -n ${NAMESPACE} -w
          """
        }
      }
    }
  }
}

发布测试环境

接收到企业微信通知后，根据通知提示的镜像版本进行发布。
测试环境的pod需要通过yaml清单手动启动，启动后才能通过Jenkins进行发布。