一、SonarQube介绍
官网:https://www.sonarsource.com/products/sonarqube/
下载地址:https://www.sonarsource.com/products/sonarqube/downloads/ 选择LTA版本,每18个月发布一次
中文插件: https://github.com/xuhuisheng/sonar-l10n-zh/tree/master scanner下载:https://docs.sonarqube.org/8.9/analysis/overview/ 插件源:https://update.sonarsource.org/ 多分支插件: https://github.com/mc1arke/sonarqube-community-branch-plugin/releases 关联提交:https://github.com/gabrie-allaigre/sonar-gitlab-plugin/tree/4.1.0-SNAPSHOT
SonarQube®是一种自动代码审查工具,可检测代码中的错误,漏洞和代码味道。它可以与您现有的工作流程集成,以实现跨项目分支和拉取请求的持续代码检查。
1.1 组件与服务组成
1.SonarQube Server启动3个主要进程:
Web服务器,供开发人员,管理人员浏览高质量的快照并配置SonarQube实例
基于Elasticsearch的Search Server从UI进行搜索服务。
Compute Engine服务器,负责处理代码分析报告并将其保存在SonarQube数据库中。
2.SonarQube数据库要存储:
SonarQube实例的配置(安全,插件设置等)项目,视图质量快照。
3.服务器上安装了多个SonarQube插件,可能包括语言,SCM,集成,身份验证和管理插件。
4.在持续集成服务器上运行一个或多个SonarScanner,以分析项目。
二、安装配置
2.1 服务端配置
## 创建数据目录 mkdir /data/sonarqube/{conf,extensions,logs,data} -p chmod 777 -R /data/sonarqube/ ## 运行 docker run -itd --name sonarqube \ -p 9000:9000 \ -v /data/sonarqube/conf:/opt/sonarqube/conf \ -v /data/sonarqube/extensions:/opt/sonarqube/extensions \ -v /data/sonarqube/logs:/opt/sonarqube/logs \ -v /data/sonarqube/data:/opt/sonarqube/data \ sonarqube:9.9.5-community ## 验证 docker logs -f sonarqube ## lib目录mkdir -p /data/sonarqube/lib cd /data/sonarqube/lib docker cp sonarqube:/opt/sonarqube/lib/* ./ docker run -itd --name sonarqube \ -p 9000:9000 \ -v /data/sonarqube/conf:/opt/sonarqube/conf \ -v /data/sonarqube/extensions:/opt/sonarqube/extensions \ -v /data/sonarqube/logs:/opt/sonarqube/logs \ -v /data/sonarqube/data:/opt/sonarqube/data \ -v /data/sonarqube/lib:/opt/sonarqube/lib \ sonarqube:9.9.5-community
下载汉化插件:
http://192.168.10.20:9000/ 默认账号密码:admin/admin
登录进入后修改默认密码---Administrator-----MarketPlace---在插件部分搜搜chinese,点击安装后,提示重启
下载的插件默认放置位置:/opt/sonarqube/extensions
手动安装插件
https://github.com/xuhuisheng/sonar-l10n-zh/tree/master
方法:
在github下载对应版本兼容的插件,放置到extensions/downloads目录下,添加+x权限,然后重启sonarqube
2.2 配置Scanner
scanner的类型有很多, 可以通过官网:https://docs.sonarsource.com/sonarqube/9.9/analyzing-source-code/overview/ 获取支持的列表。
- 变更项目代码: 可以使用构建工具进行扫描。例如maven、ant、gradle可以在配置文件中引入对应的配置。
- 不变更项目代码配置: 可以使用Jenkins或其他平台的扩展插件, 以及使用命令行进行扫描。
Anything else (CLI) - SonarScanner 通用版本下载
# 下载 cd /data https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli- 5.0.1.3006-linux.zip unzip sonar-scanner-5.0.1.3006-linux.zip # 配置环境变量 vi /etc/profile export SONAR_SCANNER_HOME=/data/sonar-scanner-5.0.1.3006-linux export PATH=$SONAR_SCANNER_HOME/bin:$PATH [root@harbor bin]# sonar-scanner -v INFO: Scanner configuration file: /data/sonar-scanner-5.0.1.3006-linux/conf/sonar-scanner.properties INFO: Project root configuration file: NONE INFO: SonarScanner 5.0.1.3006 INFO: Java 17.0.7 Eclipse Adoptium (64-bit) INFO: Linux 3.10.0-1160.71.1.el7.x86_64 amd64 # jdk版本默认自带,有可能和本机版本不一致,可以修改为和系统一致 vim bin/sonar-scanner use_embedded_jre=false
2.3 sonarscanner使用方法
代码扫描过程: 本地(构建节点)安装配置SonarScanner环境,然后通过设置sonar的一系列参数进行扫描分析。
- 配置文件方式读取扫描参数
- 命令行方式读取扫描参数
# 定义唯一的关键字,一般和项目名称一致
sonar.projectKey=devops-hello-service
# 定义项目名称
sonar.projectName=devops-hello-service
# 定义项目的版本信息,非必要定义
sonar.projectVersion=1.0
# 指定扫描代码的目录位置(多个逗号分隔)
sonar.sources=.
# 执行项目编码
sonar.sourceEncoding=UTF-8
# 指定sonar Server
sonar.host.url=
# 认证信息
sonar.login
sonar.password
-Dsonar.projectKey=xxx
。# 指定配置文件 sonar-scanner -Dproject.settings=myproject.properties # 命令行传参 sonar-scanner -Dsonar.projectKey=myproject -Dsonar.sources=src1
举例配置:
将代码下载到/data/maven目录下,在该目录下创建扫描文件
[root@harbor maven]# ll
总用量 4
-rw-r--r-- 1 root root 435 6月 3 12:46 sonar.conf
drwxr-xr-x 2 root root 6 6月 3 12:46 src
[root@harbor maven]# cat sonar.conf
# 定义唯一的关键字
sonar.projectKey=devops-hello-service
# 定义项目名称
sonar.projectName=devops-hello-service
# 定义项目的版本信息
sonar.projectVersion=1.0
# 指定扫描代码的目录位置(多个逗号分隔)
sonar.sources=./src
# 执行项目编码
sonar.sourceEncoding=UTF-8
# 指定sonar Server
sonar.host.url=http://192.168.10.20:9000
# 认证信息
sonar.login=admin
sonar.password=wg1q2w3e
# java代码需要指定编译后的类,编译后的文件在clssses目录下
sonar.java.binaries=/target/classes
执行扫描命令
# src目录是空目录
cd /data/maven
[root@harbor maven]# sonar-scanner -Dproject.settings=sonar.conf
INFO: Scanner configuration file: /data/sonar-scanner-5.0.1.3006-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: /data/maven/sonar.conf
INFO: SonarScanner 5.0.1.3006
INFO: Java 17.0.7 Eclipse Adoptium (64-bit)
INFO: Linux 3.10.0-1160.71.1.el7.x86_64 amd64
INFO: User cache: /root/.sonar/cache
INFO: Analyzing on SonarQube server 9.9.5.90363
INFO: Default locale: "zh_CN", source code encoding: "UTF-8"
INFO: Load global settings
INFO: Load global settings (done) | time=477ms
INFO: Server id: 147B411E-AY_cLgP5o0N5H03ahDoj
INFO: User cache: /root/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=320ms
INFO: Plugin [l10nzh] defines 'l10nen' as base plugin. This metadata can be removed from manifest of l10n plugins since version 5.2.
INFO: Load/download plugins (done) | time=7619ms
INFO: Process project properties
INFO: Process project properties (done) | time=23ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=8ms
INFO: Project key: devops-hello-service
INFO: Base dir: /data/maven
INFO: Working dir: /data/maven/.scannerwork
INFO: Load project settings for component key: 'devops-hello-service'
WARN: SCM provider autodetection failed. Please use "sonar.scm.provider" to define SCM of your project, or disable the SCM Sensor in the project settings.
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=670ms
INFO: Load active rules
INFO: Load active rules (done) | time=10591ms
INFO: Load analysis cache
INFO: Load analysis cache (404) | time=192ms
WARN: Property 'sonar.password' is deprecated. It will not be supported in the future. Please instead use the 'sonar.login' parameter with a token.
INFO: Load project repositories
INFO: Load project repositories (done) | time=227ms
INFO: Indexing files...
INFO: Project configuration:
INFO: 0 files indexed
INFO: ------------- Run sensors on module devops-hello-service
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=359ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=10ms
INFO: Sensor CSS Rules [javascript]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [javascript] (done) | time=2ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=1ms
INFO: Sensor C# Analysis Log [csharp]
INFO: Sensor C# Analysis Log [csharp] (done) | time=118ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=0ms
INFO: Sensor TextAndSecretsSensor [text]
INFO: Sensor TextAndSecretsSensor [text] (done) | time=30ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=2ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=45ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=1ms
INFO: Sensor IaC Docker Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Docker Sensor [iac] (done) | time=196ms
INFO: ------------- Run sensors on project
INFO: Sensor Analysis Warnings import [csharp]
INFO: Sensor Analysis Warnings import [csharp] (done) | time=2ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=0ms
INFO: SCM Publisher No SCM system was detected. You can use the 'sonar.scm.provider' property to explicitly specify it.
INFO: CPD Executor Calculating CPD for 0 files
INFO: CPD Executor CPD calculation finished (done) | time=0ms
INFO: Analysis report generated in 218ms, dir size=122.1 kB
INFO: Analysis report compressed in 22ms, zip size=14.9 kB
INFO: Analysis report uploaded in 1251ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: http://192.168.10.20:9000/dashboard?id=devops-hello-service
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.10.20:9000/api/ce/task?id=AY_cbLRw4UGYn3RQW_Mo
INFO: Analysis total time: 21.672 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 32.284s
INFO: Final Memory: 16M/64M
INFO: ------------------------------------------------------------------------
在平台上可以看到刚执行的扫描结果展示,由于项目是空分支,所以没有显示全
如果分支有代码:可以进去看扫描报告
在代码目录可以看到隐藏文件夹中扫描信息
[root@harbor maven]# cat .scannerwork/report-task.txt projectKey=devops-hello-service serverUrl=http://192.168.10.20:9000 serverVersion=9.9.5.90363 dashboardUrl=http://192.168.10.20:9000/dashboard?id=devops-hello-service ceTaskId=AY_cbLRw4UGYn3RQW_Mo ceTaskUrl=http://192.168.10.20:9000/api/ce/task?id=AY_cbLRw4UGYn3RQW_Mo
扩展Docker运行sonarscanner
docker run \ --rm \ -e SONAR_HOST_URL="http://${SONARQUBE_URL}" \ -e SONAR_LOGIN="myAuthenticationToken" \ -v "${YOUR_REPO}:/usr/src" \ sonarsource/sonar-scanner-cli
关于项目扫描参数可以参考:https://docs.sonarqube.org/latest/analysis/analysis-parameters/
可以参考官方的各种语言的扫描示例:https://docs.sonarsource.com/sonarqube/9.9/analyzing-source-code/languages/go/
Java Code Quality and Security
SonarJS SonarGO
插件,并重启服务器2.3.1 java项目扫描
sonar.projectKey
指定项目的关键字,sonar.host.url
指定服务器地址(可以直接在配置文件中写死),projectName
指定项目的名称, projectVersion
指定项目的版本(可以用构建时间和构建ID定义),login
指定登录用户名,password
指定登录用户密码, projectDescription
指定项目的描述信息, links.homepage
指定项目的主页(超链接), sources
指定扫描的目录, sourceEncoding
指定扫描时的编码, java.binaries
指定编译后的类文件目录(必填), java.test.binaries
指定编译后的测试类目录,java.surefire.report
指定测试报告目录。
sonar-scanner -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.projectKey=devops-maven-service \ -Dsonar.projectName=devops-maven-service \ -Dsonar.projectVersion=1.1 \ -Dsonar.login=admin \ -Dsonar.password=wg1q2w3e \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://192.168.10.20/devops/devops-maven-service \ -Dsonar.links.ci=http://192.168.10.20:8080/job/demo-pipeline-service/ \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports
2.3.2 web前端项目扫描
sonar-scanner \ -Dsonar.projectKey=demo-devops-ui \ -Dsonar.projectName=demo-devops-ui \ -Dsonar.sources=src \ -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.login=0809881d71f2b06b64786ae3f81a9acf22078e8b \ -Dsonar.projectVersion=2.0 \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://192.168.10.20/devops/devops-maven-service \ -Dsonar.links.ci=http://192.168.10.20:8080/job/demo-pipeline-service/ \ -Dsonar.sourceEncoding=UTF-8
2.3.3 Golang项目扫描
sonar-scanner -Dsonar.projectKey=devops-golang-service \ -Dsonar.projectName=devops-golang-service \ -Dsonar.sources=src \ -Dsonar.login=admin \ -Dsonar.password=admin \ -Dsonar.host.url=http://192.168.10.20:9000 ## 有测试用例的情况 sonar.exclusions=**/*_test.go sonar.tests=. sonar.test.inclusions=**/*_test.go
三、CI流水线集成
集成方式:1. 使用命令行方式 2. 使用Jenkins扩展插件的方式。
3.1 命令行方式
流水线中添加代码扫描阶段, 然后在script
标签中定义一段脚本。 (其实这段脚本就是我们手动在服务器上面执行的sonar-scanner的命令和参数组成的)【可以先运行该段代码确保扫描成功,然后进一步优化】
stage("SonarScan"){ steps{ script{ sh """ sonar-scanner \ -Dsonar.projectKey=demo-devops-service \ -Dsonar.projectName=demo-devops-service \ -Dsonar.sources=src \ -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.login=0809881d71f2b06b64786ae3f81a9acf22078e8b \ -Dsonar.projectVersion=1.0 \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://192.168.10.20/devops/devops-maven-service \ -Dsonar.links.ci=http://192.168.10.20:8080/job/demo-pipeline-service/ \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports """ } } } }
通过上面的代码可以发现一些问题:
- 扫描参数值都是写死的,无法使其他项目通用。
- 代码中存在敏感数据信息。
既然想通用,就要制定规范。 例如:
- 统一使用Jenkins的作业名称做为SonarQube项目名称。
- 将Sonar在进行扫描时用到的认证信息,也存储到Jenkins的系统凭据中(Secret Text类型),避免流水线中存在敏感信息。然后使用
withCredentials
将凭据的值内容赋值给变量SONAR_TOKEN
引用。 - 使用
BUILD_NUMBER
作为sonarqube项目版本(可以使用时间戳、版本号、commitid)。 -Dsonar.links.ci=${BUILD_URL}
SonarQube的扩展链接, 方便在系统中跳转。-Dsonar.links.homepage=${env.srcUrl}
SonarQube的扩展链接, 方便在系统中跳转。
优化后:
// 凭据列表
credentials = ["sonar" : '06bf5ee4-f571-4fe4-9b52-d17190ce54e5']
//服务器列表
servers = ["sonar": 'http://192.168.1.200:9000']
pipeline {
...
stage("CodeScan"){
steps{
script {
withCredentials([string(credentialsId: "${credentials['sonar']}", variable: 'SONAR_TOKEN')]) {
sh """
sonar-scanner \
-Dsonar.projectKey=${JOB_NAME.split('/')[-1]} \
-Dsonar.projectName=${JOB_NAME.split('/')[-1]} \
-Dsonar.sources=src \
-Dsonar.host.url=${servers['sonar']} \
-Dsonar.login=${SONAR_TOKEN} \
-Dsonar.projectVersion=${BUILD_NUMBER} \
-Dsonar.ws.timeout=30 \
-Dsonar.projectDescription="my first project!" \
-Dsonar.links.homepage=${env.srcUrl} \
-Dsonar.links.ci=${BUILD_URL} \
-Dsonar.sourceEncoding=UTF-8 \
-Dsonar.java.binaries=target/classes \
-Dsonar.java.test.binaries=target/test-classes \
-Dsonar.java.surefire.report=target/surefire-reports
"""
}
}
}
}
}
- 根据不同的构建工具,使用不同的代码扫描参数;
- 例如: maven 对应java类型项目, npm对应前端类型项目
withCredentials([string(credentialsId: "${credentials['sonar']}", variable: 'SONAR_TOKEN')]) { switch(buildType) { case "mvn": sh """ sonar-scanner \ -Dsonar.projectKey=${JOB_NAME.split('/')[-1]} \ -Dsonar.projectName=${JOB_NAME.split('/')[-1]} \ -Dsonar.sources=src \ -Dsonar.host.url=${servers['sonar']} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.srcUrl} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports """ break case "npm": sh """ sonar-scanner \ -Dsonar.projectKey=${JOB_NAME.split('/')[-1]} \ -Dsonar.projectName=${JOB_NAME.split('/')[-1]} \ -Dsonar.sources=src \ -Dsonar.host.url=${servers['sonar']} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.srcUrl} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sourceEncoding=UTF-8 \ """ break default: println("buildTools choice error![mvn|npm]") break; } }
最终将上述代码,纳入共享库。创建sonar.groovy。
## sonar.groovy package org.devops // 代码扫描 def SonarScan(credentials,buildType,servers){ withCredentials([string(credentialsId: "${credentials['sonar']}", variable: 'SONAR_TOKEN')]) { switch(buildType) { case "mvn": sh """ sonar-scanner \ -Dsonar.projectKey=${JOB_NAME.split('/')[-1]} \ -Dsonar.projectName=${JOB_NAME.split('/')[-1]} \ -Dsonar.sources=src \ -Dsonar.host.url=${servers['sonar']} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.srcUrl} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports """ break case "npm": sh """ sonar-scanner \ -Dsonar.projectKey=${JOB_NAME.split('/')[-1]} \ -Dsonar.projectName=${JOB_NAME.split('/')[-1]} \ -Dsonar.sources=src \ -Dsonar.host.url=${servers['sonar']} \ -Dsonar.login=${SONAR_TOKEN} \ -Dsonar.projectVersion=${BUILD_NUMBER} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=${env.srcUrl} \ -Dsonar.links.ci=${BUILD_URL} \ -Dsonar.sourceEncoding=UTF-8 \ """ break default: println("buildTools choice error![mvn|npm]") break; } } }
Jenkinsfile内容:
def sonar = new org.devops.sonar() stage("CodeScan"){ steps{ script{ sonar.SonarScan(credentials,buildType,servers) } } }
3.2 插件方式
参考:https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-jenkins/
创建SonaQube的账户token
将token保存到Jenkins凭据中
在Jenkins中安装插件sonarqube scanner。
转到"管理Jenkins>系统配置",向下滚动到SonarQube配置部分,单击Add SonarQube,添加服务器,选择凭据。
在片段生成器中查看用法, 注入与所选SonarQube 安装相关的环境变量。将设置以下变量:
SONAR_HOST_URL ## 在jenkins管理页面配置的sonar地址
SONAR_AUTH_TOKEN ## 在jenkins管理页面配置的sonar认证信息
如果此处模版有问题:
使用withSonarQubeEnv DSL引入在Jenkins中配置的sonar环境。
## 括号中的`mysonar`一定要与Jenkins设置页面定义的一致。 stage("SonarScan"){ steps { script{ groupName = "${JOB_NAME}".split("/")[0] projectName = "${JOB_NAME}".split("/")[-1] //devops03/devops03-maven-service //sonar.CodeSonar("${env.buildTool}", projectName, groupName ) withSonarQubeEnv("mysonar") { sh """ sonar-scanner -Dsonar.host.url=${SONAR_HOST_URL} \ -Dsonar.projectKey=${projectName} \ -Dsonar.projectName=${projectName} \ -Dsonar.projectVersion=${BUILD_ID} \ -Dsonar.login=${SONAR_AUTH_TOKEN} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://192.168.1.200/${groupName}/${projectName} \ -Dsonar.links.ci=http://192.168.1.200:8080/job/${groupName}/job/${projectName}/ \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 """ } } }
FAQ: sonar服务器名称错误,需要与系统设置中配置的一致。
ERROR: SonarQube installation defined in this job (mysonarserver) does not match any configured installation. Number of installations that can be configured: 1. If you want to reassign jobs to a different SonarQube installation, check the documentation under https://redirect.sonarsource.com/plugins/jenkins.html
sonar.groovy
文件中。同时定义了SonarScanWithPlugin
(插件扫描) 和SonarScan
(命令行方式扫描)。 用法没有太大的区别,使用插件扫描的共享库代码部分,可以参考使用命令行方式的讲解四、SonarQube REST API实践
SonarQube系统的API文档: http://192.168.10.20:9000/web_api
//查找项目 api/projects/search?projects=${projectName}" //创建项目 api/projects/create?name=${projectName}&project=${projectName}" //更新语言规则集 api/qualityprofiles/add_project?language=${language}&qualityProfile=${qualityProfile}&project=${projectName}" //项目授权 api/permissions/apply_template?projectKey=${projectKey}&templateName=${templateName}" //更新质量阈 api/qualitygates/select?projectKey=${projectKey}&gateId=${gateId}"
SonarQube API的请求方法
curl --location \ --request GET \ 'http://192.168.10.20:9000/api/projects/search?projects=day4-maven2-service' \ --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM0'
YWRtaW46YWRtaW4xMjM0
存储到Jenkins凭据中(Secret Text类型),后续使用withCredentials
将值赋值给变量SONAR_TOKEN
考虑到Api的URL都具有相同部分http://192.168.10.20:9000/api
所以单独复制给变量sonarApi
。每个接口返回的都是JSON类型的数据, 这里使用readJSON进行解析和处理。【所以有了下面的代码】
def SonarRequest(apiUrl,method){ withCredentials([string(credentialsId: "52df4ad9-7167-4bf6-a1fc-2f9f17713472", variable: 'SONAR_TOKEN')]) { sonarApi = "http://192.168.1.200:9000/api" response = sh returnStdout: true, script: """ curl --location \ --request ${method} \ "${sonarApi}/${apiUrl}" \ --header "Authorization: Basic ${SONAR_TOKEN}" """ try { response = readJSON text: """ ${response - "\n"} """ } catch(e){ response = readJSON text: """{"errors" : true}""" } return response } }
4.1 查找项目
接口地址和参数: http://192.168.10.20:9000/api/projects/search?projects=day4-maven2-service
请求类型: GET,可以使用postman调试
curl --location --request GET 'http://192.168.10.20:9000/api/projects/search?projects=day4-maven2-service' \ --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM0'
Jenkins Pipeline
// 查找项目 def ProjectSearch(projectName){ apiUrl = "projects/search?projects=${projectName}" response = SonarRequest(apiUrl,"GET") if (response.paging.total == 0){ println("Project not found!.....") return false } return true }
4.2 创建项目
curl --location --request POST 'http://192.168.10.20:9000/api/projects/create?name=day4-maven100-service&project=day4-maven100-service' \ --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM0'
Jenkins Pipeline
// 创建项目 def CreateProject(projectName){ apiUrl = "projects/create?name=${projectName}&project=${projectName}" response = SonarRequest(apiUrl,"POST") try{ if (response.project.key == projectName ) { println("Project Create success!...") return true } }catch(e){ println(response.errors) return false } }
4.3 更新项目质量配置
curl --location --request POST 'http://192.168.10.20:9000/api/qualityprofiles/add_project?language=java&project=day4-maven5-service&qualityProfile=devops' \ --header 'Authorization: Basic YWRtaW46YWRtaW4xMjM0'
Jenkins Pipeline
// 更新质量阈 def UpdateQualityProfiles(lang, projectName, profileName){ apiUrl = "qualityprofiles/add_project?language=${lang}&project=${projectName}&qualityProfile=${profileName}" response = SonarRequest(apiUrl,"POST") if (response.errors != true){ println("ERROR: UpdateQualityProfiles ${response.errors}...") return false } else { println("SUCCESS: UpdateQualityProfiles ${lang} > ${projectName} > ${profileName}" ) return true } }
4.4 逻辑控制
- 项目没有配置质量, 默认使用sonarway(内置质量)。
- 默认直接使用sonarscanner,扫描的项目,使用的内置的默认质量。
如果具有单独的质量配置,例如每个组织一个质量配置。 此时就需要先手动在web页面创建一个空的项目,然后在项目的配置中设置目标质量配置。(下面是手动的操作步骤)
自动化实现方式 :
- 查询项目是否存在
- 存在: 直接更新质量配置
- 不存在: 创建空的项目然后更新质量配置
stage("sonartest"){ steps{ script{ // 判断项目是否存在 sonarProjectName = "${JOB_NAME.split('/')[-1]}" result = ProjectSearch(sonarProjectName) println(result) if (result != true){ println("Create Sonar Project!...") CreateProject(sonarProjectName) } // 指定项目的配置 UpdateQualityProfiles("java", sonarProjectName, "${sonarProjectName.split('-')[0]}") } } }
完整的jenkinsfile代码
pipeline { agent { label "build" } stages{ stage("sonartest"){ steps{ script{ // 判断项目是否存在 sonarProjectName = "${JOB_NAME.split('/')[-1]}" result = ProjectSearch(sonarProjectName) println(result) if (result != true){ println("Create Sonar Project!...") CreateProject(sonarProjectName) } // 指定项目的配置 UpdateQualityProfiles("java", sonarProjectName, "${sonarProjectName.split('-')[0]}") } } } } } def SonarRequest(apiUrl,method){ withCredentials([string(credentialsId: "52df4ad9-7167-4bf6-a1fc-2f9f17713472", variable: 'SONAR_TOKEN')]) { sonarApi = "http://192.168.1.200:9000/api" response = sh returnStdout: true, script: """ curl --location \ --request ${method} \ "${sonarApi}/${apiUrl}" \ --header "Authorization: Basic ${SONAR_TOKEN}" """ try { response = readJSON text: """ ${response - "\n"} """ } catch(e){ response = readJSON text: """{"errors" : true}""" } return response } } // 更新质量阈 def UpdateQualityProfiles(lang, projectName, profileName){ apiUrl = "qualityprofiles/add_project?language=${lang}&project=${projectName}&qualityProfile=${profileName}" response = SonarRequest(apiUrl,"POST") if (response.errors != true){ println("ERROR: UpdateQualityProfiles ${response.errors}...") return false } else { println("SUCCESS: UpdateQualityProfiles ${lang} > ${projectName} > ${profileName}" ) return true } } // 创建项目 def CreateProject(projectName){ apiUrl = "projects/create?name=${projectName}&project=${projectName}" response = SonarRequest(apiUrl,"POST") try{ if (response.project.key == projectName ) { println("Project Create success!...") return true } }catch(e){ println(response.errors) return false } } // 查找项目 def ProjectSearch(projectName){ apiUrl = "projects/search?projects=${projectName}" response = SonarRequest(apiUrl,"GET") if (response.paging.total == 0){ println("Project not found!.....") return false } return true }
五、规则的启用与禁用
目的: 掌握默认规则中的一部分规则如何激活和禁用。
进入质量配置页面, 可以看到所有的语言规则配置。在这里可以看到规则的使用情况
将规则集授权给项目
使用规则: 先在页面配置项目,然后使用SonarScanner扫描。
六、其它配置
6.1、质量阀的配置
6.2 代码覆盖率统计
Maven集成Jacoco
添加jacoco-maven-plugin
和junit
插件。
<dependencies> <dependency> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.2</version> <scope>test</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> </dependencies>
添加插件
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.6.1</version> <configuration> <skipMain>true</skipMain> <skip>true</skip> <source>1.8</source> <target>1.8</target> </configuration> </plugin> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.7.5.201505241946</version> <executions> <execution> <id>prepare-agent</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>report</id> <phase>prepare-package</phase> <goals> <goal>report</goal> </goals> </execution> <execution> <id>post-unit-test</id> <phase>test</phase> <goals> <goal>report</goal> </goals> <configuration> <dataFile>target/jacoco.exec</dataFile> <outputDirectory>target/jacoco-reports</outputDirectory> </configuration> </execution> </executions> <configuration> <systemPropertyVariables> <jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile> </systemPropertyVariables> </configuration> </plugin>
# 指定代码覆盖率工具为jacoco sonar.core.codeCoveragePlugin=jacoco # 指定exec二进制文件存放路径 sonar.jacoco.reportPaths=target/jacoco.exec
cd devops-jacoco-service/ sonar-scanner -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.projectKey=devops-jacoco-service \ -Dsonar.projectName=devops-jacoco-service \ -Dsonar.projectVersion=1.0 \ -Dsonar.login=admin \ -Dsonar.password=admin \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://www.baidu.com \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec
6.3 多分支代码扫描
https://github.com/mc1arke/sonarqube-community-branch-plugin/releases/download/1.14.0/sonarqube-community-branch-plugin-1.14.0.jar
extensions/plugins
和 lib/common
目录中,然后重启sonar## 临时方案 docker exec -it sonarqube bash cd /opt/sonarqube/lib/common cp ../../extensions/plugins/sonarqube-community-branch-plugin-1.14.0 ./ exit docker restart sonarqube ## 持久化lib目录后 [root@zeyang-nuc-service sonarqube]# ls sonarqube_conf sonarqube_data sonarqube_extensions sonarqube_lib sonarqube_logs [root@zeyang-nuc-service sonarqube]# cp /root/sonarqube-community-branch-plugin-1.14.0.jar sonarqube_extensions/plugins/ [root@zeyang-nuc-service sonarqube]# chmod +x sonarqube_extensions/plugins/sonarqube-community-branch-plugin-1.14.0.jar [root@zeyang-nuc-service sonarqube]# [root@zeyang-nuc-service sonarqube]# [root@zeyang-nuc-service sonarqube]# cp /root/sonarqube-community-branch-plugin-1.14.0.jar sonarqube_lib/common/ [root@zeyang-nuc-service sonarqube]# chmod +x sonarqube_lib/common/sonarqube-community-branch-plugin-1.14.0.jar [root@zeyang-nuc-service sonarqube]# docker restart sonarqube
##扫描参数增加 –Dsonar.branch.name= sonar-scanner -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.projectKey=devops-maven2-service \ -Dsonar.projectName=devops-maven2-service \ -Dsonar.projectVersion=1.0 \ -Dsonar.login=admin \ -Dsonar.password=admin \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://192.168.10.20/devops/devops-maven-service \ -Dsonar.links.ci=http://192.168.10.20:8080/job/demo-pipeline-service/ \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.branch.name=release-1.1.1
ERROR: Error during SonarScanner execution ERROR: No branches currently exist in this project. Please scan the main branch without passing any branch parameters. ERROR: ERROR: Re-run SonarScanner using the -X switch to enable full debug logging.
6.3.1 Sonar 8.9.1 版本(备用记录)
- 将插件下载到extensions/plugins/目录。
- 更新sonar服务端的配置文件。
- 重启docker restart sonarqube 。
# cd /data/cicd2/sonarqube/ # ls sonarqube_conf sonarqube_data sonarqube_extensions sonarqube_logs # cat sonarqube_conf/sonar.properties sonar.web.javaAdditionalOpts=-javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.8.1.jar=web sonar.ce.javaAdditionalOpts=-javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.8.1.jar=ce # ls sonarqube_extensions/plugins/ sonar-gitlab-plugin-4.1.0-SNAPSHOT.jar sonar-l10n-zh-plugin-8.9.jar sonarqube-community-branch-plugin-1.8.0.jar
# 原文件描述 1. Copy the plugin JAR file to the extensions/plugins/ directory of your SonarQube instance 2. Add -javaagent:./extensions/plugins/sonarqube-community-branch-plugin-${version}.jar=web to the sonar.web.javaAdditionalOptions property in your Sonarqube installation's config/sonar.properties file, e.g. sonar.web.javaAdditionalOpts=-javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.8.0.jar=web 3. Add -javaagent:./extensions/plugins/sonarqube-community-branch-plugin-${version}.jar=ce to the sonar.ce.javaAdditionalOptions property in your Sonarqube installation's config/sonar.properties file, e.g. sonar.ce.javaAdditionalOpts=-javaagent:./extensions/plugins/sonarqube-community-branch-plugin-1.8.0.jar=ce 4. Start Sonarqube, and accept the warning about using third-party plugins
6.4 扫描结果关联commitid
提前装好插件:https://github.com/gabrie-allaigre/sonar-gitlab-plugin/tree/4.1.0-SNAPSHOT插件的说明文档查看该插件的Readme文档。 (仅质量阈失败后才可以展示扫描报告)
# cp sonar-gitlab-plugin-4.1.0-SNAPSHOT.jar /data/cicd/sonarqube/sonarqube_extensions/plugins/ # chmod +x /data/cicd/sonarqube/sonarqube_extensions/plugins/sonar-gitlab-plugin-4.1.0-SNAPSHOT.jar # docker restart sonarqube
sonar-scanner -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.projectKey=devops-jacoco-service \ -Dsonar.projectName=devops-jacoco-service \ -Dsonar.projectVersion=1.0 \ -Dsonar.login=0809881d71f2b06b64786ae3f81a9acf22078e8b \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription="my first project!" \ -Dsonar.links.homepage=http://www.baidu.com \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=coverage/jacoco.exec \ -Dsonar.gitlab.commit_sha=f898a9fdbd319e68d519aa2ff42ad80da5186103 \ -Dsonar.gitlab.ref_name=main \ -Dsonar.gitlab.project_id=37 \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=commit-status \ -Dsonar.gitlab.url=http://192.168.10.20 \ -Dsonar.gitlab.user_token=CwmDA_4TKevDPRh4_SEf \ -Dsonar.gitlab.api_version=v4
6.5 控制代码扫描步骤运行
stage("SonarScan"){ when { environment name: 'skipSonar', value: 'false' } }
七、GitLabCI-实践
import os import requests import json import sys class SonarQube(object): def __init__(self, project_name, lang, profile_name, cmds): self.server_api = "http://192.168.10.20:9000/api/" self.auth_token = "YWRtaW46YWRtaW4xMjM=" self.project_name = project_name self.lang = lang self.profile_name = profile_name self.cmds = cmds def http_req(self, method, apiUrl): url = self.server_api + apiUrl payload={} headers = { 'Authorization': 'Basic ' + self.auth_token } response = requests.request(method, url, headers=headers, data=payload) print(response.text) if response.text != "": data = json.loads(response.text) return data return {} def SearchProject(self): """查找项目""" url = "projects/search?projects=" + self.project_name response = self.http_req("GET", url) if response["paging"]["total"] == 0: return False return True def CreateProject(self): """创建项目""" apiUrl = "projects/create?name={0}&project={1}".format(self.project_name, self.project_name) response = self.http_req("POST", apiUrl) try: if response["project"]["key"] == self.project_name: return True except Exception as e : print(e) print(response["errors"]) return False def UpdateQualityProfiles(self): apiUrl = "qualityprofiles/add_project?language={0}&project={1}&qualityProfile={2}".format( self.lang, self.project_name, self.profile_name) response = self.http_req("POST", apiUrl) try : print("ERROR: UpdateQualityProfiles{0}...".format(response["errors"])) return False except Exception as e : print(e) print("SUCCESS: UpdateQualityProfiles {0} > {1} > ${2}".format( self.lang, self.project_name, self.profile_name)) return True def SonarScan(self): result = os.system(self.cmds) if result == 0: return True return False def run(self): if not self.SearchProject(): self.CreateProject() self.UpdateQualityProfiles() return self.SonarScan() if __name__ == '__main__': lang = sys.argv[1] profile_name = sys.argv[2] CI_PROJECT_NAME, CI_COMMIT_SHA, SONAR_AUTH_TOKEN,CI_PROJECT_TITLE,CI_PROJECT_URL,CI_PIPELINE_URL,CI_COMMIT_REF_NAME,CI_PROJECT_ID,CI_SERVER_URL ,GITLAB_ADMIN_TOKEN = sys.argv[3:] print(CI_PROJECT_NAME) sonarcmds = """ sonar-scanner \ -Dsonar.host.url=http://192.168.10.20:9000 \ -Dsonar.projectKey={0} \ -Dsonar.projectName={0} \ -Dsonar.projectVersion={1} \ -Dsonar.login={2} \ -Dsonar.ws.timeout=30 \ -Dsonar.projectDescription={3} \ -Dsonar.links.homepage={4} \ -Dsonar.links.ci={5} \ -Dsonar.sources=src \ -Dsonar.sourceEncoding=UTF-8 \ -Dsonar.java.binaries=target/classes \ -Dsonar.java.test.binaries=target/test-classes \ -Dsonar.java.surefire.report=target/surefire-reports \ -Dsonar.core.codeCoveragePlugin=jacoco \ -Dsonar.jacoco.reportPaths=target/jacoco.exec \ -Dsonar.gitlab.commit_sha={1} \ -Dsonar.gitlab.ref_name={6} \ -Dsonar.gitlab.project_id={7} \ -Dsonar.dynamicAnalysis=reuseReports \ -Dsonar.gitlab.failure_notification_mode=nothing \ -Dsonar.gitlab.url={8} \ -Dsonar.gitlab.user_token={9} \ -Dsonar.gitlab.api_version=v4 """.format( CI_PROJECT_NAME, CI_COMMIT_SHA, SONAR_AUTH_TOKEN, CI_PROJECT_TITLE, CI_PROJECT_URL, CI_PIPELINE_URL, CI_COMMIT_REF_NAME, CI_PROJECT_ID, CI_SERVER_URL, GITLAB_ADMIN_TOKEN) result = SonarQube(CI_PROJECT_NAME, lang, profile_name, sonarcmds ).run() print(result)
脚本调用
python3 sonarqube.py \ "java" "devops03" "devops-test" "99d098ef066b79d577a98220a17959465f4dd750" "9e7e39a14a96bc886fdde43388b91e810491b7dc" "devops" "http://192.168.10.20/devops/devops-maven-service" "http://192.168.10.20:8080/job/demo-pipeline-service/" "master" "5" "http://192.168.10.20" "apF1R9s9JJBYJzLF5mYd"
gitlabCI.yaml
include: - project: 'devops03/devops03-gitlabci-lib' ref: main file: - '/jobs/CI.yaml' workflow: rules: - if: $CI_PIPELINE_SOURCE == "web" when: always - if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000" when: never - when: always variables: GIT_CHECKOUT: "false" ## 全局关闭作业代码下载 BUILD_SHELL: "mvn clean package -DskipTests -s settings.xml" ## 构建命令 TEST_SHELL: "mvn test -s settings.xml" ## 测试命令 ARTIFACT_PATH: "target/*.jar" ## 制品路径 TEST_REPORTS: "target/surefire-reports/TEST-*.xml" ## 测试报告 stages: - build - test - sonarscan pipelineInit: extends: - .pipelineInit cibuild: extends: - .cibuild citest: extends: - .citest sonarscan: tags: - build stage: sonarscan script: |- curl "http://192.168.10.20/devops03/devops03-gitlabci-lib/-/raw/main/utils/SonarQube.py" \ -o sonarqube.py -s python sonarqube.py "java" ${CI_PROJECT_ROOT_NAMESPACE} ${CI_PROJECT_NAME} ${CI_COMMIT_SHA} \ ${SONAR_AUTH_TOKEN} ${CI_PROJECT_TITLE} ${CI_PROJECT_URL} ${CI_PIPELINE_URL} ${CI_COMMIT_REF_NAME} \ ${CI_PROJECT_ID} ${CI_SERVER_URL} ${GITLAB_ADMIN_TOKEN}
引用文档:https://www.yuque.com/devopsgroup/51ctodevops/ugh7eq?singleDoc#pPfns