DEVOPS技术实践_05：sonar静态代码扫描

一、SonarQube静态代码扫描平台

1.1 安装

https://www.sonarqube.org/官网

1.2 下载软件包

https://www.sonarqube.org/downloads/

[root@sonar-server ~]# mkdir /usr/local/sonarc

[root@sonar-server ~]# cd /usr/local/sonarc

[root@sonar-server sonarc]# wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-6.7.7.zip

[root@sonar-server sonarc]# wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-3.3.0.1492-linux.zip

[root@sonar-server sonarc]# ll

-rw-r--r--. 1 root root 159921852 Apr 17  2019 sonarqube-6.7.7.zip
-rw-r--r--. 1 root root  73866903 Jan  8 22:27 sonar-scanner-cli-3.3.0.1492-linux.zip

sonar是一个用于代码质量管理的开放平台。通过插件机制，sonar可以集成不同的测试工具，代码分析工具，以及持续集成工具。比如pmd-cpd，checkstyle，findbugs。Jkens。通过不同的插件对这些结果进行再加工处理。通过量化的方式度量代码质量的变化，从而可以方便的对不同规模和种类的工程进行代码质量管理。同时sonar还对大量的集成工具提供了接口支持，可以很方便的在持续集成中使用Sonar。

[root@sonar-server sonarc]# yum -y install java-1.8.0-openjdk-devel.x86_64

官方文档

环境要求

https://docs.sonarqube.org/latest/requirements/requirements/

JDK JRE：1.8

[root@sonar-server sonarc]# java -version

openjdk version "1.8.0_201"
OpenJDK Runtime Environment (build 1.8.0_201-b09)
OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)

1.3 安装mysql 5.6 +

[root@sonar-server sonarc]# mkdir /usr/local/mysql

[root@sonar-server sonarc]# cd /usr/local/mysql

[root@sonar-server mysql]# wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm

[root@sonar-server mysql]# rpm -ivh mysql-community-release-el7-5.noarch.rpm

Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-release-el7-5    ################################# [100%]

[root@sonar-server mysql]# yum -y install mysql mysql-devel mysql-server mysql-utilities

Installed:
  mysql-community-client.x86_64 0:5.6.43-2.el7                                        
  mysql-community-devel.x86_64 0:5.6.43-2.el7                                         
  mysql-community-libs.x86_64 0:5.6.43-2.el7                                          
  mysql-community-server.x86_64 0:5.6.43-2.el7                                        
  mysql-utilities.noarch 0:1.6.5-1.el7

启动mysql

[root@sonar-server mysql]# systemctl start mysqld

[root@sonar-server mysql]# netstat -ntlp

tcp6       0      0 :::3306                 :::*                    LISTEN      53978/mysqld

1.4 创建数据库

[root@sonar-server mysql]# mysql -u root

mysql> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+

mysql> create database sonar default CHARSET utf8;

mysql> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sonar              |
+--------------------+

mysql> grant all on sonar.* to sonar@'%' IDENTIFIED by 'meiyoumima';

mysql> flush PRIVILEGES;

1.5 安装sonar

If you're running on Linux, you must ensure that:
vm.max_map_count is greater or equals to 262144
fs.file-max is greater or equals to 65536
the user running SonarQube can open at least 65536 file descriptors
the user running SonarQube can open at least 2048 threads
You can see the values with the following commands:
sysctl vm.max_map_count
sysctl fs.file-max
ulimit -n
ulimit -u

根据文档执行命令

[root@sonar-server ~]# sysctl vm.max_map_count
vm.max_map_count = 65530
[root@sonar-server ~]# sysctl fs.file-max
fs.file-max = 379752
[root@sonar-server ~]# ulimit -n
1024
[root@sonar-server ~]# ulimit -u
15012
修改这些参数
[root@sonar-server ~]# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
[root@sonar-server ~]# sysctl -w fs.file-max=65536
fs.file-max = 65536
[root@sonar-server ~]# ulimit -n 65536
[root@sonar-server ~]# ulimit -u 2048

添加sonar用户（sonar明确规定不允许使用root运行）

[root@sonar-server ~]# useradd sonar

[root@sonar-server ~]# cd /usr/local/sonarc/

[root@sonar-server sonarc]# unzip sonarqube-6.7.7.zip -d /home/sonar/

[root@sonar-server sonarc]# cd /home/sonar/

[root@sonar-server sonar]# mv sonarqube-6.7.7 sonarqube

修改sonar的配置文件

[root@sonar-server sonar]# cd sonarqube

[root@sonar-server sonarqube]# ll

drwxr-xr-x. 8 root root  136 Apr 16  2019 bin
drwxr-xr-x. 2 root root   50 Apr 16  2019 conf
-rw-r--r--. 1 root root 7651 Apr 16  2019 COPYING
drwxr-xr-x. 2 root root   24 Apr 16  2019 data
drwxr-xr-x. 7 root root  150 Apr 16  2019 elasticsearch
drwxr-xr-x. 4 root root   40 Apr 16  2019 extensions
drwxr-xr-x. 9 root root  140 Apr 16  2019 lib
drwxr-xr-x. 2 root root    6 Apr 16  2019 logs
drwxr-xr-x. 2 root root   24 Apr 16  2019 temp
drwxr-xr-x. 9 root root 4096 Apr 16  2019 web

[root@sonar-server sonarqube]# ll ./bin/

drwxr-xr-x. 2 root root  25 Apr 16  2019 jsw-license
drwxr-xr-x. 3 root root  48 Apr 16  2019 linux-x86-32
drwxr-xr-x. 3 root root  48 Apr 16  2019 linux-x86-64
drwxr-xr-x. 3 root root  48 Apr 16  2019 macosx-universal-64
drwxr-xr-x. 3 root root 167 Apr 16  2019 windows-x86-32
drwxr-xr-x. 3 root root 167 Apr 16  2019 windows-x86-64

[root@sonar-server sonarqube]# ll ./bin/linux-x86-64/

drwxr-xr-x. 2 root root     27 Apr 16  2019 lib
-rwxr-xr-x. 1 root root  15522 Apr 16  2019 sonar.sh     #sonar的启动脚本
-rwxr-xr-x. 1 root root 111027 Apr 16  2019 wrapper

[root@sonar-server sonarqube]# vim conf/sonar.properties

sonar.web.host=172.25.254.133
sonar.web.port=9000
sonar.jdbc.username=sonar
sonar.jdbc.password=meiyoumima
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance&useSSL=false

修改文件的属主属组

[root@sonar-server sonar]# chown -R sonar:sonar /home/sonar/*

切换用户设置环境变量

[root@sonar-server sonar]# su - sonar

[sonar@sonar-server ~]$ vim .bash_profile

export SONAR_HOME=/home/sonar/sonarqube
export PATH=$PATH:$SONAR_HOME/bin

[sonar@sonar-server ~]$ source .bash_profile

[sonar@sonar-server ~]$ ./sonarqube/bin/linux-x86-64/sonar.sh start

Starting SonarQube...
Started SonarQube.

1.7 启动sonar

[sonar@sonar-server ~]$ ./sonarqube/bin/linux-x86-64/sonar.sh

Usage: ./sonarqube/bin/linux-x86-64/sonar.sh { console | start | stop | restart | status | dump }

[sonar@sonar-server ~]$ ./sonarqube/bin/linux-x86-64/sonar.sh console

Running SonarQube...     #输出信息
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
jvm 1    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
jvm 1    | 
jvm 1    | 2019.04.13 15:17:58 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /home/sonar/sonarqube/temp
jvm 1    | 2019.04.13 15:17:58 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
jvm 1    | 2019.04.13 15:17:58 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/home/sonar/sonarqube/elasticsearch]: /home/sonar/sonarqube/elasticsearch/bin/elasticsearch -Epath.conf=/home/sonar/sonarqube/temp/conf/es
jvm 1    | 2019.04.13 15:17:58 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
jvm 1    | 2019.04.13 15:18:14 INFO  app[][o.e.p.PluginsService] no modules loaded
jvm 1    | 2019.04.13 15:18:14 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
jvm 1    | 2019.04.13 15:18:52 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
jvm 1    | 2019.04.13 15:18:52 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/home/sonar/sonarqube]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/home/sonar/sonarqube/temp -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -cp ./lib/common/*:./lib/server/*:/home/sonar/sonarqube/lib/jdbc/mysql/mysql-connector-java-5.1.42.jar org.sonar.server.app.WebServer /home/sonar/sonarqube/temp/sq-process7782197132334949329properties

看日志

[sonar@sonar-server ~]$ cd sonarqube/logs/
[sonar@sonar-server logs]$ tail -f es.log
2019.04.13 15:19:22 INFO  es[][o.e.n.Node] closing ...
2019.04.13 15:19:22 INFO  es[][o.e.n.Node] closed
[sonar@sonar-server logs]$ tail -f sonar.log 
2019.04.13 15:19:22 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.04.13 15:19:22 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
[sonar@sonar-server logs]$ tail -n 100 web.log 
    at org.sonar.db.DefaultDatabase.checkConnection(DefaultDatabase.java:106)    #数据库问题，不能使用sonar@localhost
    ... 29 common frames omitted
Caused by: java.sql.SQLException: Access denied for user 'sonar'@'localhost' (using password: YES)

尝试使用sonar连接数据库

[sonar@sonar-server logs]$ mysql -hlocalhost -usonar -pmeiyoumima
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'sonar'@'localhost' (using password: YES)
[sonar@sonar-server logs]$ mysql -h127.0.0.1 -usonar -pmeiyoumima
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'sonar'@'localhost' (using password: YES)
[sonar@sonar-server logs]$ mysql -h172.25.254.133 -usonar -pmeiyoumima    #成功
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| sonar |
+--------------------+

修改配置文件连接数据库

[sonar@sonar-server logs]$ vim /home/sonar/sonarqube/conf/sonar.properties

sonar.web.host=172.25.254.133
sonar.web.port=9000
sonar.jdbc.username=sonar
sonar.jdbc.password=meiyoumima
sonar.jdbc.url=jdbc:mysql://172.25.254.133:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance&useSSL=false

[sonar@sonar-server logs]$ /home/sonar/sonarqube/bin/linux-x86-64/sonar.sh start

[sonar@sonar-server logs]$ /home/sonar/sonarqube/bin/linux-x86-64/sonar.sh console

Running SonarQube...
SonarQube is already running.

成功

1.8 浏览器访问

初始密码：admin admin登录

1.9 安装一个maketpalce的汉化插件

重启

二、配置snoar-scanner

2.1 安装

[sonar@sonar-server logs]$ unzip /usr/local/sonarc/sonar-scanner-cli-3.3.0.1492-linux.zip -d /home/sonar/

[sonar@sonar-server logs]$ cd

[sonar@sonar-server ~]$ ll

drwxr-xr-x. 11 sonar sonar 141 Apr 16  2019 sonarqube
drwxr-xr-x.  6 sonar sonar  51 Jan  8 12:19 sonar-scanner-3.3.0.1492-linux

[sonar@sonar-server ~]$ mv sonar-scanner-3.3.0.1492-linux sonar-scanner

[sonar@sonar-server ~]$ cd sonar-scanner/

[sonar@sonar-server sonar-scanner]$ ll

drwxr-xr-x. 2 sonar sonar  54 Jan  8 12:19 bin
drwxr-xr-x. 2 sonar sonar  38 Jan  8 12:17 conf
drwxr-xr-x. 4 sonar sonar 186 Jan  8 12:19 jre
drwxr-xr-x. 2 sonar sonar  46 Jan  8 12:19 lib

[sonar@sonar-server sonar-scanner]$ vim conf/sonar-scanner.properties

sonar.host.url=http://172.25.254.133:9000
sonar.sourceEncoding=UTF-8

[root@sonar-server ~]# vim .bash_profile

export SCAN_HOME=/home/sonar/sonar-scanner/
export PATH=$PATH:$SCAN_HOME/bin

[root@sonar-server ~]# source .bash_profile

2.2 测试

[root@sonar-server ~]# sonar-scanner -X

[root@sonar-server ~]# mkdir code

[root@sonar-server ~]# cd code/

[root@sonar-server code]# vim sonar-project.properties

sonar.projectKey=test-project1
sonar.projectName=cloud
sonar.projectVersion=1.0
sonar.source=src
sonar.language=python
sonar.sourceEncoding=UTF-8

[root@sonar-server code]mkdir src

[root@sonar-server code]# vim ./src/test.py

print("HelloWorld")
print("HelloWorld")
print("HelloWorld")
print("HelloWorld")
print("HelloWorld")
print("HelloWorld")
print("HelloWorld")
print("HelloWorld")

执行

[root@sonar-server code]# python ./src/test.py
HelloWorld
HelloWorld
HelloWorld
HelloWorld
HelloWorld
HelloWorld
HelloWorld
HelloWorld

[root@sonar-server ~]# sonar-scanner

ERROR: You must define the following mandatory properties for 'Unknown': sonar.projectKey, sonar.sources
ERROR: 
ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

2.3 修改权限

[root@sonar-server code]# chown sonar:sonar /root/code/* -R

[root@sonar-server code]# ll

-rw-r--r--. 1 sonar sonar 485 Apr 21 18:23 sonar-project.properties

[root@sonar-server ~]# sonar-scanner

ERROR: Error during SonarQube Scanner execution
ERROR: No quality profiles have been found, you probably don't have any language plugin installed.
ERROR: 
ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

2.4 安装plugin

[root@sonar-server code]# sonar-scanner

ERROR: Error during SonarQube Scanner execution
ERROR: You must install a plugin that supports the language 'python'
ERROR: #依然报错
ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

把python修改为py

[root@sonar-server code]# vim sonar-project.properties

sonar.projectKey=test-project1
sonar.projectName=cloud
sonar.projectVersion=1.0
sonar.language=py     #查阅修改此参数   https://www.cnblogs.com/ckat/p/3638887.html
sonar.sources=src
sonar.sourceEncoding=UTF-8

[root@sonar-server code]# sonar-scanner

INFO: Scanner configuration file: /home/sonar/sonar-scanner-3.3.0.1492-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: /root/code/sonar-project.properties
INFO: SonarQube Scanner 3.3.0.1492
INFO: Java 1.8.0_121 Oracle Corporation (64-bit)
INFO: Linux 3.10.0-693.el7.x86_64 amd64
INFO: User cache: /root/.sonar/cache
INFO: SonarQube server 6.7.7
INFO: Default locale: "en_US", source code encoding: "UTF-8"
INFO: Publish mode
INFO: Load global settings
INFO: Load global settings (done) | time=62ms
INFO: Server id: A623D34D-AWoVn6_8P1KovjAYWYot
INFO: User cache: /root/.sonar/cache
INFO: Load plugins index
INFO: Load plugins index (done) | time=112ms
INFO: Plugin [l10nzh] defines 'l10nen' as base plugin. This metadata can be removed from manifest of l10n plugins since version 5.2.
INFO: Process project properties
INFO: Load project repositories
INFO: Load project repositories (done) | time=35ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=29ms
INFO: Load active rules
INFO: Load active rules (done) | time=344ms
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=26ms
INFO: Project key: test-project1
INFO: -------------  Scan cloud
INFO: Load server rules
INFO: Load server rules (done) | time=34ms
INFO: Base dir: /root/code
INFO: Working dir: /root/code/.scannerwork
INFO: Source paths: src
INFO: Source encoding: UTF-8, default locale: en_US
INFO: Language is forced to py
INFO: Index files
INFO: 1 file indexed
INFO: Quality profile for py: Sonar way
INFO: Sensor Python Squid Sensor [python]
INFO: Sensor Python Squid Sensor [python] (done) | time=101ms
INFO: Sensor Cobertura Sensor for Python coverage [python]
INFO: Sensor Cobertura Sensor for Python coverage [python] (done) | time=11ms
INFO: Sensor PythonXUnitSensor [python]
INFO: Sensor PythonXUnitSensor [python] (done) | time=0ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=28ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: SCM Publisher is disabled
INFO: 1 file had no CPD blocks
INFO: Calculating CPD for 0 files
INFO: CPD calculation finished
INFO: Analysis report generated in 274ms, dir size=6 KB
INFO: Analysis reports compressed in 15ms, zip size=3 KB
INFO: Analysis report uploaded in 410ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://172.25.254.133:9000/dashboard/index/test-project1
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://172.25.254.133:9000/api/ce/task?id=AWo_l71TD8zrSa_Nq7yJ
INFO: Task total time: 2.315 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 3.566s
INFO: Final Memory: 9M/183M
INFO: ------------------------------------------------------------------------