Spring Cloud第九篇 | 分布式服务跟踪Sleuth

本文是Spring Cloud专栏的第九篇文章，了解前八篇文章内容有助于更好的理解本文：

一、Sleuth前言

随着业务的发展,系统规模也会变得越来越大,各微服务间的调用关系也变得越来越错综复杂。通常一个由客户端发起的请求在后端系统中会经过多个不同的微服务调用来协同产生最后的请求结果,在复杂的微服务架构系统中,几乎每一个前端请求都会形成一条复杂的分布式服务调用链路,在每条链路中任何一个依赖服务出现延迟过高或错误的时候都有可能引起请求最后的失败。这时候, 对于每个请求,全链路调用的跟踪就变得越来越重要,通过实现对请求调用的跟踪可以帮助我们快速发现错误根源以及监控分析每条请求链路上的性能瓶颈等。

Spring Cloud组件中Spring Cloud Sleuth提供了一套完整的解决方案，下面将介绍Spring Cloud Sleuth的应用

二、Sleuth快速入门

1、为了保持其他模块的整洁性，重新搭建一个消费者(springcloud-consumer-sleuth)，提供者(springcloud-consumer-sleuth)，消费者和提供者都是和前面所用的都一样没有什么区别，注册中心还是使用前面案例的注册中心(springcloud-eureka-server/8700)，详细查看案例源码。

2、完成以上工作之后，我们为服务提供者和服务消费者添加跟踪功能，通过Spring Cloud Sleuth的封装，我们为应用增加服务跟踪能力的操作非常方便，只需要在服务提供者和服务消费者增加spring-cloud-starter-sleuth依赖即可

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

3、访问消费者接口，然后查看控制台日志显示

消费者(springcloud-consumer-sleuth)打印的日志

2019-12-05 12:30:20.178  INFO [springcloud-consumer-sleuth,f6fb983680aab32b,f6fb983680aab32b,false] 8992 --- [nio-9090-exec-1] c.s.controller.SleuthConsumerController  : === consumer hello ===

提供者(springcloud-provider-sleuth)打印的日志

2019-12-05 12:30:20.972  INFO [springcloud-provider-sleuth,f6fb983680aab32b,c70932279d3b3a54,false] 788 --- [nio-8080-exec-1] c.s.controller.SleuthProviderController  : === provider hello ===

从上面的控制台输出内容中,我们可以看到多了一些形如 [springcloud-consumer-sleuth,f6fb983680aab32b,c70932279d3b3a54,false]的日志信息,而这些元素正是实现分布式服务跟踪的重要组成部分,每个值的含义如下所述：

第一个值: springcloud-consumer-sleuth,它记录了应用的名称,也就是application properties 中spring.application.name参数配置的属性
第二个值:f6fb983680aab32b, Spring Cloud Sleuth生成的一个ID,称为Trace ID, 它用来标识一条请求链路。一条请求链路中包含一个Trace ID,多个Span ID
第三个值:c70932279d3b3a54, Spring Cloud Sleuth生成的另外一个ID,称为Span ID,它表示一个基本的工作单元,比如发送一个HTTP请求
第四个值: false,表示是否要将该信息输出到Zipkin等服务中来收集和展示。上面四个值中的Trace ID和Span ID是Spring Cloud Sleuth实现分布式服务跟踪的核心，在一次服务请求链路的调用过程中,会保持并传递同一个Trace ID,从而将整个分布于不同微服务进程中的请求跟踪信息串联起来。以上面输出内容为例springcloud-consumer-sleuth和springcloud-provider-sleuth同属于一个前端服务请求资源，所以他们的Trace ID是相同的，处于同一条请求链路中。

三、跟踪原理

分布式系统中的服务跟踪在理论上并不复杂,主要包括下面两个关键点：

为了实现请求跟踪,当请求发送到分布式系统的入口端点时,只需要服务跟踪框架为该请求创建一个唯一的跟踪标识,同时在分布式系统内部流转的时候,框架始终保持传递该唯一标识,直到返回给请求方为止,这个唯一标识就是前文中提到的Trace ID。通过Trace ID的记录,我们就能将所有请求过程的日志关联起来
为了统计各处理单元的时间延迟,当请求到达各个服务组件时,或是处理逻辑到达某个状态时,也通过一个唯一标识来标记它的开始、具体过程以及结束,该标识就是前面提到的Span ID。对于每个Span来说,它必须有开始和结束两个节点,通过记录开始Span和结束Span的时间戳,就能统计出该Span的时间延迟,除了时间戳记录之外,它还可以包含一些其他元数据,比如事件名称、请求信息等

在【二、sleuth快速入门】示例中,我们轻松实现了日志级别的跟踪信息接入,这完全归功于spring-cloud-starter-sleuth组件的实现，在SpringBoot应用中通过在工程中引入spring-cloud-starter-sleuth依赖之后，他会自动为当前应用构建起各通信通道的跟踪机制，比如：

通过RabbitMQ、Kafka(或者其他任何Spring Cloud Stream绑定器实现的消息中间件)传递的请求
通过Zuul代理传递的请求
通过RestTemplate发起的请求

在【二、sleuth快速入门】示例中,由于springcloud-consumer-sleuth对springcloud-provider-sleuth发起的请求是通过RestTemplate实现的,所以spring-cloud-starter-sleuth组件会对该请求进行处理。在发送到springcloud-provider-sleuth之前,Sleuth会在该请求的Header中增加实现跟踪需要的重要信息,主要有下面这几个(更多关于头信息的定义可以通过查看org.springframework.cloud.sleuth.Span的源码获取)。

X-B3-TraceId:一条请求链路( Trace)的唯一标识,必需的值。
X-B3- SpanId:一个工作单元(Span)的唯一标识,必需的值。
X-B3- ParentSpanId:标识当前工作单元所属的上一个工作单元, Root Span(请求链路的第一个工作单元)的该值为空。
X-B3-Sampled:是否被抽样输出的标志，1表示需要被输出，0表示不需要被输出。
X-B3-Name:工作单元的名称

可以通过对springcloud-provider-sleuth的实现做一些修改来输出这些头信息，具体如下：

private final Logger logger = Logger.getLogger(SleuthProviderController.class.getName());

    @RequestMapping("/hello")
    public String hello(HttpServletRequest request){
        logger.info("=== provider hello ===,Traced={"+request.getHeader("X-B3-TraceId")+"},SpanId={"+request.getHeader("X-B3- SpanId")+"}");
        return "Trace";
    }

通过上面的改造，再次重启案例，然后访问我们查看日志，可以看到提供者输出了正在处理的TraceId和SpanId信息。

消费者(springcloud-consumer-sleuth)打印的日志

2019-12-05 13:15:01.457  INFO [springcloud-consumer-sleuth,41697d7fa118c150,41697d7fa118c150,false] 10036 --- [nio-9090-exec-2] c.s.controller.SleuthConsumerController  : === consumer hello ===

提供者(springcloud-provider-sleuth)打印的日志

2019-12-05 13:15:01.865  INFO [springcloud-provider-sleuth,41697d7fa118c150,863a1245c86b580e,false] 11088 --- [nio-8080-exec-1] c.s.controller.SleuthProviderController  : === provider hello ===,Traced={41697d7fa118c150},SpanId={863a1245c86b580e}

四、抽样收集

通过Trace ID和Span ID已经实现了对分布式系统中的请求跟踪,而记录的跟踪信息最终会被分析系统收集起来,并用来实现对分布式系统的监控和分析功能,比如,预警延迟过长的请求链路、查询请求链路的调用明细等。此时,我们在对接分析系统时就会碰到个问题：分析系统在收集跟踪信息的时候,需要收集多少跟踪信息才合适呢？

理论上来说,我们收集的跟踪信息越多就可以越好地反映出系统的实际运行情况,并给出更精准的预警和分析。但是在高并发的分布式系统运行时,大量的请求调用会产生海量的跟踪日志信息，如果收集过多的跟踪信息将会对整个分布式系统的性能造成一定的影响，同时保存大量的日志信息也需要不少的存储开销。所以，在Sleuth中采用了抽象收集的方式来为跟踪信息打上收集标识，也就是我们之前在日志信息中看到的第4个布尔类型的值，他代表了该信息是否被后续的跟踪信息收集器获取和存储。

public abstract class Sampler {
  /**
   * Returns true if the trace ID should be measured.
   *
   * @param traceId The trace ID to be decided on, can be ignored
   */
  public abstract boolean isSampled(long traceId);
}

通过实现isSampled方法, Spring Cloud Sleuth会在产生跟踪信息的时候调用它来为跟踪信息生成是否要被收集的标志。需要注意的是,即使isSampled返回了false,它仅代表该跟踪信息不被输出到后续对接的远程分析系统(比如Zipkin中,对于请求的跟踪活动依然会进行,所以我们在日志中还是能看到收集标识为fase的记录。

默认情况下, Sleuth会使用SamplerProperties实现的抽样策略,以请求百分比的方式配置和收集跟踪信息。我们可以通过在application.yml中配置下面的参数对其百分比值进行设置,它的默认值为0.1,代表收集10%的请求跟踪信息。

spring:
  sleuth:
    sampler:
      probability: 0.1

在开发调试期间,通常会收集全部跟踪信息并输出到远程仓库,我们可以将其值设置为1,或者也可以注入Sampler对象SamplerProperties策略,比如

@Bean
  public Sampler defaultSampler() {
    return Sampler.ALWAYS_SAMPLE;
  }

由于跟踪日志信息数据的价值往往仅在最近一段时间内非常有用，比如一周。那么我们在设计抽样策略时，主要考虑在不对系统造成明显性能影响的情况下，以在日志保留时间窗内充分利用存储空间的原则来实现抽样策略。

五、与Zipkin整合

由于日志文件都离散地存储在各个服务实例的文件系之上，仅通过查看日志信息文件来分我们的请求链路依然是一件相当麻烦的事情，所以我们需要一些工具来帮助集中收集、存储和搜索这些跟踪信息，比如ELK日志平台，虽然通过ELK平台提供的收集、存储、搜索等强大功能,我们对跟踪信息的管理和使用已经变得非常便利。但是在ELK平台中的数据分析维度缺少对请求链路中各阶段时间延迟的关注,很多时候我们追溯请求链路的一个原因是为了找出整个调用链路中出现延迟过高的瓶颈源,或为了实现对分布式系统做延迟监控等与时间消耗相关的需求,这时候类似ELK这样的日志分析系统就显得有些乏力了。对于这样的问题,我们就可以引入Zipkin来得以轻松解决。

Zipkin是Twitter的一个开源项目,它基于Google Dapper实现。我们可以使用它来收集各个服务器上请求链路的跟踪数据,并通过它提供的REST API接口来辅助査询跟踪数据以实现对分布式系统的监控程序,从而及时发现系统中出现的延迟升高问题并找出系统性能瓶颈的根源。除了面向开发的API接口之外,它还提供了方便的UI组件来帮助我们直观地搜索跟踪信息和分析请求链路明细,比如可以査询某段时间内各用户请求的处理时间等。

下图展示了Zipkin的基础架构，他主要由4个核心组成：

Collector：收集器组件,它主要处理从外部系统发送过来的跟踪信息,将这些信息转换为 Zipkin内部处理的Span格式,以支持后续的存储、分析、展示等功能。
Storage：存储组件,它主要处理收集器接收到的跟踪信息,默认会将这些信息存储在内存中。我们也可以修改此存储策略,通过使用其他存储组件将跟踪信息存储到数据库中。
RESTful API：API组件,它主要用来提供外部访问接口。比如给客户端展示跟踪信息,或是外接系统访问以实现监控等。
Web UI：UI组件,基于AP组件实现的上层应用。通过UI组件,用户可以方便而又直观地查询和分析跟踪信息。

1、构建server-zipkin

在Spring Cloud为F版本的时候，已经不需要自己构建Zipkin Server了，只需要下载jar即可，下载地址：https://dl.bintray.com/openzipkin/maven/io/zipkin/zipkin-server/

Zipkin的github地址：https://github.com/openzipkin

2、下载完成jar 包之后，需要运行jar，如下

java -jar zipkin-server-2.10.1-exec.jar

访问浏览器http://localhost:9411，如图我们可以看到zipkin的管理界面

3、为应用引入和配置Zipkin服务

我们需要对应用做一些配置，以实现将跟踪信息输出到Zipkin Server。我们使用【二、sleuth快速入门】中实现的消费者(springcloud-consumer-sleuth)，提供者(springcloud-provider-sleuth)为例，对他们进行改造，都加入整合Zipkin的依赖

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

4、在消费者(springcloud-consumer-sleuth)，提供者(springcloud-provider-sleuth)中增加Zipkin Server的配置信息，具体信息如下所示，默认是连接地址为：http://localhost:9411

spring:
  zipkin:
    base-url: http://localhost:9411

5、测试与分析

到这里我们已经完成了配置Zipkin Server的所有基本工作，然后访问几次消费者接口http://localhost:9090/consumer/hello,当在日志中出现跟踪信息的最后一个值为true的时候，说明该跟踪信息会输出给Zipkin Server，如下日志

2019-12-05 15:47:25.600  INFO [springcloud-consumer-sleuth,cbdbbebaf32355ab,cbdbbebaf32355ab,false] 8564 --- [nio-9090-exec-9] c.s.controller.SleuthConsumerController  : === consumer hello ===
2019-12-05 15:47:27.483  INFO [springcloud-consumer-sleuth,8f332a4da3c05f62,8f332a4da3c05f62,false] 8564 --- [nio-9090-exec-6] c.s.controller.SleuthConsumerController  : === consumer hello ===
2019-12-05 15:47:42.127  INFO [springcloud-consumer-sleuth,61b922906800ac60,61b922906800ac60,true] 8564 --- [nio-9090-exec-2] c.s.controller.SleuthConsumerController  : === consumer hello ===
2019-12-05 15:47:42.457  INFO [springcloud-consumer-sleuth,1acae9ebecc4d36d,1acae9ebecc4d36d,false] 8564 --- [nio-9090-exec-4] c.s.controller.SleuthConsumerController  : === consumer hello ===
2019-12-05 15:47:42.920  INFO [springcloud-consumer-sleuth,b2db9e00014ceb88,b2db9e00014ceb88,false] 8564 --- [nio-9090-exec-7] c.s.controller.SleuthConsumerController  : === consumer hello ===
2019-12-05 15:47:43.457  INFO [springcloud-consumer-sleuth,ade4d5a7d97ca16b,ade4d5a7d97ca16b,false] 8564 --- [nio-9090-exec-9] c.s.controller.SleuthConsumerController  : === consumer hello ===

所以此时可以在Zipkin Server的管理界面中选择合适的查询条件，单击Find Traces按钮，就可以查询出刚才在日志中出现的跟踪信息了（也可以根据日志信息中的Treac ID，在页面右上角的输入框中来搜索），页面如下所示：

点击下方springcloud-consumer-sleuth端点的跟踪信息，还可以得到Sleuth 跟踪到的详细信息，其中包括我们关注的请求时间消耗等。

点击导航栏中的《依赖分析》菜单，还可以查看Zipkin Server根据跟踪信息分析生成的系统请求链路依赖关系图，如下所示

六、Zipkin将数据存储到ElasticSearch中

在【五、与Zipkin整合】中链路收集的数据默认存储在Zipkin服务的内存中，Zipkin服务一重启这些数据就没了，在开发环境中我们图方便省方便可以直接将数据存储到内存中，但是在生产环境，我们需要将这些数据持久化。我们可以将其存储在MySQL中，实际使用中数据量可能会比较大，所以MySQL并不是一种很好的选择，可以选择用Elasticsearch来存储数据，Elasticsearch在搜索方面有先天的优势。

1、上面几个步骤使用的 zipkin-server-2.10.1-exec.jar 是以前下载的，再此使用的zipkin server版本为2.19.2，下载地址：https://dl.bintray.com/openzipkin/maven/io/zipkin/zipkin-server/

2、zipkin-server-2.19.2-exec.jar 版本仅支持Elasticsearch5-7.x版本，注意版本对应，自行上elastic官网下载安装Elasticsearch5-7.x版本，ES服务准备就绪完成之后。

3、启动zipkin服务命令如下：

java -DSTORAGE_TYPE=elasticsearch -DES_HOSTS=http://47.112.11.147:9200 -jar zipkin-server-2.19.2-exec.jar

另外还有一些其它可配置参数，具体参考：https://github.com/openzipkin/zipkin/tree/master/zipkin-server#elasticsearch-storage

* `ES_HOSTS`: A comma separated list of elasticsearch base urls to connect to ex. http://host:9200.
Defaults to "http://localhost:9200".
* `ES_PIPELINE`: Indicates the ingest pipeline used before spans are indexed. No default.
* `ES_TIMEOUT`: Controls the connect, read and write socket timeouts (in milliseconds) for
Elasticsearch Api. Defaults to 10000 (10 seconds)
* `ES_INDEX`: The index prefix to use when generating daily index names. Defaults to zipkin.
* `ES_DATE_SEPARATOR`: The date separator to use when generating daily index names. Defaults to '-'.
* `ES_INDEX_SHARDS`: The number of shards to split the index into. Each shard and its replicas
are assigned to a machine in the cluster. Increasing the number of shards
and machines in the cluster will improve read and write performance. Number
of shards cannot be changed for existing indices, but new daily indices
will pick up changes to the setting. Defaults to 5.
* `ES_INDEX_REPLICAS`: The number of replica copies of each shard in the index. Each shard and
its replicas are assigned to a machine in the cluster. Increasing the
number of replicas and machines in the cluster will improve read
performance, but not write performance. Number of replicas can be changed
for existing indices. Defaults to 1. It is highly discouraged to set this
to 0 as it would mean a machine failure results in data loss.
* `ES_USERNAME` and `ES_PASSWORD`: Elasticsearch basic authentication, which defaults to empty string.
Use when X-Pack security (formerly Shield) is in place.
* `ES_HTTP_LOGGING`: When set, controls the volume of HTTP logging of the Elasticsearch Api.
Options are BASIC, HEADERS, BODY

4、我们修改springcloud-provider-sleuth，springcloud-consumer-sleuth的application.yml文件将抽样概率修改为1，方便测试

spring:
  sleuth:
    sampler:
      probability: 1

5、然后访问http://localhost:9090/consumer/hello接口几次，再次访问kibana可以看到索引已经创建了

6、可以看到里面已经存储数据了

7、访问zipkin可以看到信息

8、但是依赖中没有任何信息

9、zipkin会在ES中创建以zipkin开头日期结尾的索引，并且默认以天为单位分割，使用ES存储模式时，zipkin中的依赖信息会无法显示，通过zipkin官网可以看到，我们需要通过zipkin-dependencies工具包计算

10、zipkin-dependencies生成依赖链

zipkin-dependencies基于spark job来生成全局的调用链，此处下载

zipkin-dependencies的版本为2.4.1

github地址：https://github.com/openzipkin/zipkin-dependencies

下载地址：https://dl.bintray.com/openzipkin/maven/io/zipkin/dependencies/zipkin-dependencies/

11、下载完成之后启动

这个jar包就不要再windows上启动了，启动不了，启动到你怀疑人生。在linux上执行

官方网文档给了个Linux案例：

STORAGE_TYPE=cassandra3 java -jar zipkin-dependencies.jar `date -u -d '1 day ago' +%F`

STORAGE_TYPE为存储类型，我门这里使用的是ES所以修改为elasticsearch，后面的date参数命令可以用来显示或设定系统的日期与时间，不了解的自行百度。

启动命令为：

ZIPKIN_LOG_LEVEL=DEBUG ES_NODES_WAN_ONLY=true STORAGE_TYPE=elasticsearch  ES_HOSTS=http://47.112.11.147:9200  java -Xms256m -Xmx1024m -jar zipkin-dependencies-2.4.1.jar `date -u -d '1 day ago' +%F`

下载完成后通过上述命令启动zipkin-dependencies，这里要注意的是程序只会根据当日的zipkin数据实时计算一次依赖关系，我们是昨天(2019-12-17)收集到ES的数据，所以今天 (2019-12-18)我们在启动命令中指定前一天，就能生成依赖数据以索引zipkin:dependency-2019-12-17方式存入ES中，然后就退出了(Done)，因此要做到实时更新依赖的话需要周期性执行zipkin-dependencies，例如使用Linux中的crontab定时调度等等。

执行后日志如下：

[root@VM_0_8_centos local]# ZIPKIN_LOG_LEVEL=DEBUG ES_NODES_WAN_ONLY=true STORAGE_TYPE=elasticsearch  ES_HOSTS=http://47.112.11.147:9200  java -Xms256m -Xmx1024m -jar zipkin-dependencies-2.4.1.jar `date -u -d '1 day ago' +%F`
19/12/18 21:44:10 WARN Utils: Your hostname, VM_0_8_centos resolves to a loopback address: 127.0.0.1; using 172.21.0.8 instead (on interface eth0)
19/12/18 21:44:10 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: spark.ui.enabled=false
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: es.index.read.missing.as.empty=true
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: es.nodes.wan.only=true
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: es.net.ssl.keystore.location=
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: es.net.ssl.keystore.pass=
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: es.net.ssl.truststore.location=
19/12/18 21:44:10 DEBUG ElasticsearchDependenciesJob: Spark conf properties: es.net.ssl.truststore.pass=
19/12/18 21:44:10 INFO ElasticsearchDependenciesJob: Processing spans from zipkin:span-2019-12-17/span
19/12/18 21:44:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/12/18 21:44:12 WARN Java7Support: Unable to load JDK7 types (annotations, java.nio.file.Path): no Java7 support added
19/12/18 21:44:13 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:13 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:13 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:16 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:16 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:16 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:16 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:17 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:17 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:17 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:17 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:17 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:17 WARN Resource: Detected type name in resource [zipkin:span-2019-12-17/span]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:18 DEBUG DependencyLinker: building trace tree: traceId=a5253479e359638b
19/12/18 21:44:18 DEBUG DependencyLinker: traversing trace tree, breadth-first
19/12/18 21:44:18 DEBUG DependencyLinker: processing {"traceId":"a5253479e359638b","id":"a5253479e359638b","kind":"SERVER","name":"get /consumer/hello","timestamp":1576591155280041,"duration":6191,"localEndpoint":{"serviceName":"springcloud-consumer-sleuth","ipv4":"192.168.0.104"},"remoteEndpoint":{"ipv6":"::1","port":62085},"tags":{"http.method":"GET","http.path":"/consumer/hello","mvc.controller.class":"SleuthConsumerController","mvc.controller.method":"hello"}}
19/12/18 21:44:18 DEBUG DependencyLinker: root's client is unknown; skipping
19/12/18 21:44:18 DEBUG DependencyLinker: processing {"traceId":"a5253479e359638b","parentId":"a5253479e359638b","id":"8d6b8fb1bbb4f48c","kind":"CLIENT","name":"get","timestamp":1576591155281192,"duration":3999,"localEndpoint":{"serviceName":"springcloud-consumer-sleuth","ipv4":"192.168.0.104"},"tags":{"http.method":"GET","http.path":"/provider/hello"}}
19/12/18 21:44:18 DEBUG DependencyLinker: processing {"traceId":"a5253479e359638b","parentId":"a5253479e359638b","id":"8d6b8fb1bbb4f48c","kind":"SERVER","name":"get /provider/hello","timestamp":1576591155284040,"duration":1432,"localEndpoint":{"serviceName":"springcloud-provider-sleuth","ipv4":"192.168.0.104"},"remoteEndpoint":{"ipv4":"192.168.0.104","port":62182},"tags":{"http.method":"GET","http.path":"/provider/hello","mvc.controller.class":"SleuthProviderController","mvc.controller.method":"hello"},"shared":true}
19/12/18 21:44:18 DEBUG DependencyLinker: found remote ancestor {"traceId":"a5253479e359638b","parentId":"a5253479e359638b","id":"8d6b8fb1bbb4f48c","kind":"CLIENT","name":"get","timestamp":1576591155281192,"duration":3999,"localEndpoint":{"serviceName":"springcloud-consumer-sleuth","ipv4":"192.168.0.104"},"tags":{"http.method":"GET","http.path":"/provider/hello"}}
19/12/18 21:44:18 DEBUG DependencyLinker: incrementing link springcloud-consumer-sleuth -> springcloud-provider-sleuth
19/12/18 21:44:18 DEBUG DependencyLinker: building trace tree: traceId=54af196ac59ee13e
19/12/18 21:44:18 DEBUG DependencyLinker: traversing trace tree, breadth-first
19/12/18 21:44:18 DEBUG DependencyLinker: processing {"traceId":"54af196ac59ee13e","id":"54af196ac59ee13e","kind":"SERVER","name":"get /consumer/hello","timestamp":1576591134958091,"duration":139490,"localEndpoint":{"serviceName":"springcloud-consumer-sleuth","ipv4":"192.168.0.104"},"remoteEndpoint":{"ipv6":"::1","port":62085},"tags":{"http.method":"GET","http.path":"/consumer/hello","mvc.controller.class":"SleuthConsumerController","mvc.controller.method":"hello"}}
19/12/18 21:44:18 DEBUG DependencyLinker: root's client is unknown; skipping
19/12/18 21:44:18 DEBUG DependencyLinker: processing {"traceId":"54af196ac59ee13e","parentId":"54af196ac59ee13e","id":"1a827ae864bd2399","kind":"CLIENT","name":"get","timestamp":1576591134962066,"duration":133718,"localEndpoint":{"serviceName":"springcloud-consumer-sleuth","ipv4":"192.168.0.104"},"tags":{"http.method":"GET","http.path":"/provider/hello"}}
19/12/18 21:44:18 DEBUG DependencyLinker: processing {"traceId":"54af196ac59ee13e","parentId":"54af196ac59ee13e","id":"1a827ae864bd2399","kind":"SERVER","name":"get /provider/hello","timestamp":1576591135064214,"duration":37707,"localEndpoint":{"serviceName":"springcloud-provider-sleuth","ipv4":"192.168.0.104"},"remoteEndpoint":{"ipv4":"192.168.0.104","port":62089},"tags":{"http.method":"GET","http.path":"/provider/hello","mvc.controller.class":"SleuthProviderController","mvc.controller.method":"hello"},"shared":true}
19/12/18 21:44:18 DEBUG DependencyLinker: found remote ancestor {"traceId":"54af196ac59ee13e","parentId":"54af196ac59ee13e","id":"1a827ae864bd2399","kind":"CLIENT","name":"get","timestamp":1576591134962066,"duration":133718,"localEndpoint":{"serviceName":"springcloud-consumer-sleuth","ipv4":"192.168.0.104"},"tags":{"http.method":"GET","http.path":"/provider/hello"}}
19/12/18 21:44:18 DEBUG DependencyLinker: incrementing link springcloud-consumer-sleuth -> springcloud-provider-sleuth
19/12/18 21:44:18 INFO ElasticsearchDependenciesJob: Saving dependency links to zipkin:dependency-2019-12-17/dependency
19/12/18 21:44:18 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:18 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:18 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:18 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 WARN Resource: Detected type name in resource [zipkin:dependency-2019-12-17/dependency]. Type names are deprecated and will be removed in a later release.
19/12/18 21:44:19 INFO ElasticsearchDependenciesJob: Processing spans from zipkin-span-2019-12-17
19/12/18 21:44:20 INFO ElasticsearchDependenciesJob: No dependency links could be processed from spans in index zipkin-span-2019-12-17
19/12/18 21:44:20 INFO ElasticsearchDependenciesJob: Done

12、上面有一个配置：ES_NODES_WAN_ONLY=true

Whether the connector is used against an Elasticsearch instance in a cloud/restricted environment over the WAN, such as Amazon Web Services. In this mode, the connector disables discovery and only connects through the declared es.nodes during all operations, including reads and writes. Note that in this mode, performance is highly affected.

该配置的含义为通过公网我访问云上或者一些限制性网络上的ES实例时，如AWS，通过声明该配置就会禁用发现其它节点的行为，后续的读和写都只会通过这个指定的节点进行操作，增加了该属性就可以访问云上或者受限制网络中的ES，但是也因为读写都是通过这个节点，因而性能上会受到比较大的影响。zipkin-dependencies的github上也就简单说明，如果这个配置为true，将仅使用在ES_HOSTS主机中设置的值，例如ES集群在Docker中。

13、查看kibana中生成的索引