Jaeger demo 应用: HotROD 分析

HotROD 是 Jaeger 官方的 demo 应用，分析这一应用如何使用 Jaeger Client API 可以窥见 Jaeger 最佳实践的"样子"。

HotROD 源码：https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod

部署环境

部署 Jaeger Operator

使用 Jaeger Operator 来部署 Jaeger 实例可以省去很多不必要的麻烦。

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm upgrade --install jaeger-operator jaegertracing/jaeger-operator -n jaeger-operator --create-namespace

部署 Jaeger 实例

使用 all-in-one 的方式部署一个 Jaeger 实例，这样 trace 数据只驻留在内存中，重启 pod 会丢失 trace 数据。但这对于分析 demo 应用足够了。

kubectl apply -n jaeger-operator -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: all-in-one
spec:
  strategy: allinone
  agent:
    strategy: DaemonSet
  ingress:
    hosts:
    - jaeger.k8s.local
EOF

部署 HotROD

为 HotROD 应用创建 Pod、Service 和 Ingress：

kubectl apply -n jaeger-operator -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hotrod
spec:
  rules:
  - host: hotrod.jaeger.k8s.local
    http:
      paths:
      - backend:
          service:
            name: hotrod
            port:
              number: 8080
        path: /
        pathType: Prefix
---
apiVersion: v1
kind: Service
metadata:
  name: hotrod
  labels:
    app: hotrod
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: hotrod
---
apiVersion: v1
kind: Pod
metadata:
  name: hotrod
  labels:
    app: hotrod
spec:
  containers:
  - name: hotrod
    image: jaegertracing/example-hotrod
    imagePullPolicy: IfNotPresent
    args:
    - all
    - --jaeger-ui=http://jaeger.k8s.local
    env:
    - name: JAEGER_AGENT_HOST
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
EOF

测试

浏览器打开 http://hotrod.jaeger.k8s.local，点击页面四个按钮的任意一个，便会发起服务调用。服务调用的结果会显示在页面上，附带一个 find trace 链接，点击转跳到 jaeger 的 web ui。

HotROD 构架

包含4个微服务：frontend、customer、driver、route

2个数据服务：mysql、redis

服务调用关系如下：

graph TB B("浏览器") F["frontend"] C["customer"] D["driver"] R["route"] RD["redis"] MS["mysql"] B -- http--> F F -- http --> C --> MS F -- grpc --> D --> RD F -- http --> R

HotROD 消息流

sequenceDiagram participant B as 浏览器 participant F as frontend participant C as customer participant D as driver participant R as route participant MS as mysql participant RD as redis B ->>+ F: http request: /dispatch F ->>+ C: http request: /customer C ->>+ MS: select MS ->>- C: mysql response C ->>- F: http response F ->>+ D: grpc request: FindNearest D ->>+ RD: FindDriverIDs RD ->>- D: redis response D ->>+ RD: GetDriver * N RD ->>- D: redis response D ->>- F: grpc response F ->>+ R: http request: /route * N R ->>- F: http response F ->>- B: http response

HTTP 服务端 Span 埋点

主要在 nethttp.Middleware 实现

sequenceDiagram participant S as http.Server participant MF as nethttp.MiddlewareFunc participant HF as http.HandlerFunc S ->>+ MF: call(req) note right of MF: <--<br/>ctx = tracer.Extract(req.HTTPHeaders)<br/>span = tracer.StartSpan(ctx)<br/>req.Ctx = ContextWithSpan(req.Ctx, span)) MF ->>+ HF: call(req) HF ->>- MF: return note right of MF: <-- span.Finish() MF ->>- S: return

HTTP 客户端 Span 埋点

主要在 nethttp.Transport 实现

sequenceDiagram participant A as Application participant C as http.Client participant T as nethttp.Transport participant RT as http.RoundTripper note right of A: <--<br/>req.Ctx = ... <br/>req, ht = TraceRequest(tracer, req) A ->>+ C: Do(req) C ->>+ T: RoundTrip(req) note right of T: <--<br/>parentSpan = SpanFromContext(req.Ctx)<br/>ht.root = tracer.StartSpan(ChildOf(parentSpan.Ctx))<br/>ht.sp = tracer.StartSpan(ChildOf(ht.root.Ctx))<br/>carrier = HTTPHeadersCarrier(req.Header)<br/>ht.sp.Tracer().Inject(ht.sp.Ctx, carrier) T ->>+ RT: RoundTrip(req) RT ->>- T: return rsp note right of T: <-- rsp.Body = closeTracker{rsp.Body, ht.sp} T ->>- C: return rsp C ->>- A: return rsp note right of A: <-- ht.root.Finish() note right of A: <--<br>... do stuffs ...<br/>rsp.Body.Close() => ht.sp.Finish()

注意：这里有两个span——ht.root 和 ht.sp，ht.root 在 rsp 返回时 finish，ht.sp 在 rsp.Body 关闭时 finish。在 http multipart response 的情况下，收到 rsp 时起 body 可能还不完整。

GRPC 服务端 Span 埋点

在 grpc.NewServer 时传入 OpenTracingServerInterceptor 和 OpenTracingStreamServerInterceptor 实现。

细节和 http 服务端大同小异。

GRPC 客户端 Span 埋点

在 grpc.Dial 时传入 OpenTracingClientInterceptor 和 OpenTracingStreamClientInterceptor 实现。

细节和 http 客户端大同小异。