skywalking系列(2)-在IntelliJ IDEA中通过skywalking源码构建项目
目的
为了学习skywalking源码的设计，以及更好的了解内部原理，对系统进行扩展，优化。
参考
github上skywalking官方教程:https://github.com/apache/skywalking/blob/master/docs/en/guides/How-to-build.md
SkyWalking 6.x 源码分析 —— 调试环境搭建【芋道源码】:http://www.iocoder.cn/SkyWalking/6/build-debugging-environment/
环境准备
操作系统: macOS 10.14.1
skywalking源码版本: github上6.6.0之后的master分支(2020-1-29)
IDE: intelliJ IDEA
jdk: 1.8
maven: maven 3
构建项目
1.clone代码
git clone https://github.com/apache/skywalking.git
遇到的问题
(1) 问题：clone代码失败：
解决办法：配置host，可加快网速
219.76.4.4 github-cloud.s3.amazonaws.com
192.30.253.112 github.com
151.101.185.194 github.global.ssl.fastly.net
2.选择maven方式导入skywalking源码到IDEA
3.执行git命令
git submodule init
4.执行git命令
git submodule update
这两步的目的是获取skywalking子模块的源码，子模块包括apm-network、query-graphql-plugin、skywalking-ui、e2e-ttl-es，具体的依赖信息可查看/.gitmodules文件
[submodule "apm-protocol/apm-network/src/main/proto"]
path = apm-protocol/apm-network/src/main/proto
url = https://github.com/apache/skywalking-data-collect-protocol.git
[submodule "oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol"]
path = oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol
url = https://github.com/apache/skywalking-query-protocol.git
[submodule "skywalking-ui"]
path = skywalking-ui
url = https://github.com/apache/skywalking-rocketbot-ui.git
[submodule "test/e2e/e2e-ttl/e2e-ttl-es/src/main/proto"]
path = test/e2e/e2e-ttl/e2e-ttl-es/src/main/proto
url = https://github.com/apache/skywalking-data-collect-protocol.git
- 【敲黑板】该步骤非常重要，不完整执行成功，后续的编译会失败。
- git submodule update执行很慢，还可能中途中断，但是执行的过程中不要手动中断命令的执行，等10分钟看结果，不然文件可能只下载了一部分，导致莫名其妙的问题。
遇到的问题
(1)git submodule update失败
terrydeMacBook-Pro:skywalking-terry terry$ git submodule init
terrydeMacBook-Pro:skywalking-terry terry$ git submodule update
Submodule path 'oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol': checked out '6fc96650acc7f539fcdf6d51648525ce93e5fa1a'
fatal: Needed a single revision
Unable to find current revision in submodule path 'skywalking-ui'
解决办法：
参考：https://www.cnblogs.com/ligun123/p/4139883.html
删除skywalking-ui目录，再执行git submodule update，将会出现如下：
terrydembp:skywalking-github terry$ git submodule update
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 4370 (delta 2), reused 3 (delta 0), pack-reused 4356
Receiving objects: 100% (4370/4370), 13.76 MiB | 36.00 KiB/s, done.
Resolving deltas: 100% (2575/2575), done.
From https://github.com/apache/skywalking-rocketbot-ui
* [new branch] develop -> origin/develop
* [new branch] master -> origin/master
* [new tag] v0.9.0 -> v0.9.0
* [new tag] v1.0.0 -> v1.0.0
* [new tag] v1.0.1 -> v1.0.1
* [new tag] v1.0.2 -> v1.0.2
* [new tag] v1.0.3 -> v1.0.3
* [new tag] v6.1.0 -> v6.1.0
* [new tag] v6.2.0 -> v6.2.0
* [new tag] v6.3.0 -> v6.3.0
* [new tag] v6.4.0 -> v6.4.0
* [new tag] v6.5.0 -> v6.5.0
* [new tag] v6.6.0 -> v6.6.0
Submodule path 'skywalking-ui': checked out '9ad81a05a2f2969420c1f75769c2b4def42819e4'
Submodule path 'test/e2e/e2e-ttl/e2e-ttl-es/src/main/proto': checked out '7b244ff7ec350910295eee85633e02d92a6f6b1c'
5.执行mvn命令
./mvnw clean package -DskipTests
这一步执行时间长短和网速有关系，编译过两次，一次14min、一次1h
执行情况示例：
terrydembp:skywalking-github terry$ ./mvnw clean package -DskipTests
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 50710 100 50710 0 0 20994 0 0:00:02 0:00:02 --:--:-- 20997
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] apm [pom]
[INFO] apm-commons [pom]
省略许多行......
省略许多行......
省略许多行......
遇到的问题
(1) ./mvnw clean package -DskipTests无法编译
原因：开了vpn，原以为【翻qiang】会快很多，没想到开了VPN，反而下载不了需要的插件。
解决办法：关闭vpm
(2)执行失败，提示缺少部分类，例如缺少类KeyStringValuePair.java
[ERROR] /Users/terry/Gits/agent/skywalking/skywalking-github/apm-protocol/apm-network/src/main/java/org/apache/skywalking/apm/network/trace/component/command/TraceIgnoreCommand.java:[39,48] 找不到符号
[ERROR] 符号: 变量 KeyStringValuePair
[ERROR] 位置: 类 org.apache.skywalking.apm.network.trace.component.command.TraceIgnoreCommand
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :apm-network
可能是因为git submodule没有更新到submodule，解决办法是：找到上面提到的项目根目录下的/.gitmodules文件，查看submodule对应的path路径，看是否完整更新下来的源码。
可以删除这几个目录，然后执行git submodule init、git submodule update，重新更新submodule的文件。
执行成功的结果：
[INFO] configuration-consul ............................... SUCCESS [ 2.216 s]
[INFO] server-bootstrap ................................... SUCCESS [ 4.166 s]
[INFO] server-starter ..................................... SUCCESS [01:27 min]
[INFO] server-starter-es7 ................................. SUCCESS [ 8.393 s]
[INFO] apm-webapp ......................................... SUCCESS [07:15 min]
[INFO] apache-skywalking-apm .............................. SUCCESS [ 45.532 s]
[INFO] apache-skywalking-apm-es7 .......................... SUCCESS [ 22.466 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:01 h
[INFO] Finished at: 2020-01-29T13:03:28+08:00
6.最终在disk目录下生成了文件：
dist/apache-skywalking-apm-bin.tar.gz
dist/apache-skywalking-apm-bin-es7.tar.gz
这些包和官方的release版本的包是一样的，后续可以自己修改源码进行编译、构建、部署。
7.生成源码：
grpc-java and java folders in apm-protocol/apm-network/target/generated-sources/protobuf
grpc-java and java folders in oap-server/server-core/target/generated-sources/protobuf
grpc-java and java folders in oap-server/server-receiver-plugin/receiver-proto/target/generated-sources/protobuf
grpc-java and java folders in oap-server/exporter/target/generated-sources/protobuf
grpc-java and java folders in oap-server/server-configuration/grpc-configuration-sync/target/generated-sources/protobuf
antlr4 folder in oap-server/oal-grammar/target/generated-sources
根据官网找到上面的文件目录，
右键--->Mark Directory As --->Generated Rources Root
文件夹会由黄色变成蓝色
启动：
启动OapServer
执行org.apache.skywalking.oap.server.starter.OAPServerStartUp#main方法，启动server。
注意
- 6.6.0默认是用es7作为storage module，所以需要部署es7；
如果想用其他storage module，则可以修改/oap-server/server-bootstrap/src/main/resources/application.yml目录下的storage节点的配置。然后rebuild一下该文件，重新编译，在重新启动即可。
我采用的的是ES6，修改skywalking的配置如下，启动ES后，再启动OapServer
storage:
elasticsearch:
nameSpace: ${SW_NAMESPACE:""}
clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
#trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
#trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
user: ${SW_ES_USER:""}
password: ${SW_ES_PASSWORD:""}
indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
# Those data TTL settings will override the same settings in core module.
recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day
otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day
monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month
# Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests
flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
profileTaskQueryMaxSize: ${SW_STORAGE_ES_QUERY_PROFILE_TASK_SIZE:200}
advanced: ${SW_STORAGE_ES_ADVANCED:""}
# elasticsearch7:
# nameSpace: ${SW_NAMESPACE:""}
# clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
# protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
# #trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
# #trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
# user: ${SW_ES_USER:""}
# password: ${SW_ES_PASSWORD:""}
# indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
# indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
# # Those data TTL settings will override the same settings in core module.
# recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day
# otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day
# monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month
# # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
# bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests
# flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
# concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
# resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
# metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
# segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
# advanced: ${SW_STORAGE_ES_ADVANCED:""}
查看ES索引
访问http://localhost:9200/_cat/indices
查看ES索引可以检验skywalking初始化数据是否成功
启动webApp
执行org.apache.skywalking.apm.webapp.ApplicationStartUp方法
验证
访问http://127.0.0.1:8080/
启动普通java应用，验证链路追踪
配置应用的VM参数如下，再运行应用：
-Dskywalking.agent.service_name=skywalking-test-local -Dskywalking.collector.backend_service=127.0.0.1:11800 -javaagent:/Users/terry/Gits/agent/skywalking-bin/6.6.0/apache-skywalking-apm-bin/agent/skywalking-agent.jar
验证
访问一个springmvc的请求localhost:8082/a_testSysLog
参考skywalking
