跪拜 Guibai
← Back to the summary

Spring Insight: Tracing a Request from Gateway to Loyalty in One Waterfall

Using the GET /order/create request as an example, we trace it from the browser through the gateway, across the product/user/loyalty services, and use the console's call timeline to clarify instrumentation, propagation, and waterfall parent-child relationships.

Companion repositories: spring-insight · spring-insight-sca-demo


1. Introduction

When troubleshooting distributed systems, the worst thing is: I clearly called /order/create, but observability only shows some downstream interface.

Spring Insight takes a lightweight approach: add a Starter on the business side, run an insight-server alongside, and use Span + W3C traceparent to string a request into a single Trace.

This article focuses on one real chain:

Corresponding business entry: GET /order/create?userId=1&productId=1 (accessed via the gateway).

2. Build an Overall Impression with a Diagram

2.1 What the Topology Looks Like

2.2 This Entry in the Trace List

2.3 Call Timeline (Waterfall)

3. Why Does One Request Show 13 Call Timeline Entries

First, look at what the order service does in the demo - the downstream order in the waterfall almost mirrors the execution order of this code.

sca-order: OrderController#createOrder

@GetMapping("/create")
public String createOrder(Long userId, Long productId) {
    // ① Get product price
    BigDecimal price = productFeignClient.getPrice(productId);
    // ② Get current score
    Integer currentScore = userFeignClient.getScore(userId);
    // ③ Add score based on price
    Integer addScore = price.intValue();
    Integer finalScore = userFeignClient.addScore(userId, currentScore, addScore);
    return "Order created...";
}

sca-user does not calculate the score itself, but passes it through to sca-loyalty:

@GetMapping("/score/{id}")
public Integer getScore(Long id) {
    return loyaltyFeignClient.getScore(id);
}

@GetMapping("/addScore")
public Integer addScore(...) {
    return loyaltyFeignClient.addScore(...);
}

So one /order/create business-wise:

Gateway forwarding
  → Order.create
      → Product.price          (serial 1)
      → User.score → Loyalty.score   (serial 2)
      → User.addScore → Loyalty.addScore (serial 3)

In the waterfall, if "product and user run in parallel" appears, that is an anomaly; normally there should be three Feign segments one after another.

4. Top to Bottom: Why Each Step Appears

Below, explain according to the console's "call timeline" visual order (top to bottom). Durations are from one real run (magnitudes fluctuate, structure should be stable).

Row 1: sca-gateway · GET /order/create · SERVER · ~35ms

Row 2: sca-gateway · GET /order/create → sca-order · CLIENT · ~33ms

Row 3: sca-order · GET /order/create · SERVER · ~25ms

Rows 4–5: Get Product Price (Serial Segment 1)

Row Service Operation (schematically) Kind Role
4 sca-order GET sca-product/product/price/1 CLIENT Feign outbound
5 sca-product GET /product/price/1 SERVER Product inbound

Rows 6–9: Get Score (Serial Segment 2)

Row Service Operation (schematically) Kind
6 sca-order GET sca-user/user/score/1 CLIENT
7 sca-user GET /user/score/1 SERVER
8 sca-user GET sca-loyalty/loyalty/score/1 CLIENT
9 sca-loyalty GET /loyalty/score/1 SERVER

Rows 10–13: Add Score (Serial Segment 3)

Row Service Operation (schematically) Kind
10 sca-order GET sca-user/user/addScore?... CLIENT
11 sca-user GET /user/addScore SERVER
12 sca-user GET sca-loyalty/loyalty/addScore?... CLIENT
13 sca-loyalty GET /loyalty/addScore SERVER

Structure Overview (corresponding to the waterfall)

① Gateway SERVER          GET /order/create
 └─② Gateway CLIENT       GET /order/create → sca-order
     └─③ Order SERVER     GET /order/create
         ├─④ Order CLIENT → ⑤ Product SERVER     /product/price/1
         ├─⑥ Order CLIENT → ⑦ User SERVER        /user/score/1
         │                    └─⑧ User CLIENT → ⑨ Loyalty SERVER
         └─⑩ Order CLIENT → ⑪ User SERVER       /user/addScore
                              └─⑫ User CLIENT → ⑬ Loyalty SERVER

5. Implementation Breakdown: How a Span Is Born and How Parent-Child Is Attached

5.1 Role Division

Component Responsibility
insight-agent / starter Inbound SERVER, outbound CLIENT, sampling, W3C inject/extract, async batch reporting
insight-server Receives Spans, aggregates by traceId, chooses root for list, waterfall layout and critical path
Console UI Topology / list / waterfall; presentation layer distinguishes SERVER/CLIENT (does not change operationName in storage)

5.2 Inbound SERVER (Two Runtimes)

Servlet (order / product / user / loyalty)

  1. HttpRequestInterceptor.preHandle: TraceContext.startSpan(method + uri, remoteTraceId, remoteParent, …), sets spanKind=SERVER, stores the Span in the request attribute.
  2. Controller Feign / RestTemplate runs.
  3. afterCompletion: ends the Span stored on the request and reports a snapshot, then TraceContext.clear().

Key point: outbound CLIENT is not pushed onto the stack; the stack should always hold the current HTTP SERVER. If async dispatch empties ThreadLocal, the request attribute still allows the ingress span to report - this is the core fix in 0.4.1.

WebFlux / Gateway

5.3 Outbound CLIENT (Feign / RestTemplate / Gateway)

Common:

  1. parent = TraceContext.currentSpan() (or the exchange's SERVER in Gateway).
  2. new TraceSpan(parent.traceId, parent.spanId) → this CLIENT's spanId becomes the parent for the downstream traceparent.
  3. spanKind=CLIENT, remoteService uses a logical service name when possible (lb:// / Feign Target / non-IP Host), to avoid topology edges becoming 127.0.0.1.
  4. After the call, finish + reportSpan(snapshot), not pushed onto the TraceContext stack.

This shapes the classic "CLIENT wraps downstream SERVER" in the waterfall.

5.4 W3C traceparent: The Glue Across Processes

When spring.insight.http-trace-propagation-enabled (default true) is on:

traceparent: 00-<trace-id>-<parent-span-id>-<flags>

Without propagation, each service would start its own Trace, and the topology and waterfall would break.

5.5 How the Server Draws the Waterfall

  1. Fetch all Spans by traceId.
  2. Build a tree using parentSpanId to compute indentation depth; those without a parent (cross-service entry) are treated as "local root".
  3. Sort by startTime, draw bars by relative offset.
  4. List "root operation name" priority: no parent → parent not in this Trace → earliest start; when tied, prefer SERVER.
  5. UI adds badges and → remoteService to solve the hard-to-distinguish Gateway SERVER/CLIENT with the same name.

6. Two Pitfalls (Encountered in This Chain)

Pitfall A: List Root Becomes sca-product/...

Phenomenon: You accessed /order/create, but the most recent request shows the product CLIENT.

Cause: The order's ingress SERVER was not stored, so the list picks the earliest Span among those "parent not in this Trace" - which happens to be the first Feign CLIENT.

Fix direction: The inbound completion path must always report the SERVER attached to the request/exchange (Agent 0.4.1).

Pitfall B: The First Two Waterfall Rows Both Show GET /order/create

Phenomenon: Looks like a duplicate request.

Cause: The gateway's inbound SERVER and outbound CLIENT intentionally share the same path name (the real HTTP path is this).

Display optimization: badge distinguishes Kind; CLIENT appends → sca-order (UI only, not changing reported fields).

7. Reproduce This Chain Yourself

  1. Start insight-server (e.g., ghcr.io/iweidujiang/spring-insight-server).
  2. Start spring-insight-sca-demo https://github.com/iweidujiang/spring-insight-sca-demo (contains gateway / order / product / user / loyalty).
  3. Use Agent ≥ 0.4.1 (includes the ingress SERVER reporting fix).
  4. Browser access: http://localhost:8080/order/create?userId=1&productId=1 (actual port per demo).
  5. Open console → Trace → click that Trace, compare with Section 4.

8. Summary

Spring Insight deliberately does not build a second full APM: making HTTP inbound/outbound + W3C + one adequate waterfall solid is enough to explain "the order chain".

If JDBC/MQ are added later, they will extend the same "CLIENT attaches to SERVER, propagation attaches across processes" model.

Finally, the repository is here:

https://github.com/iweidujiang/spring-insight

Please give a Star, file Issues.

I really want stars.jpg

Comments

Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.

卷土的土

Lost entries can cause the root node to be misjudged.

苏渡苇

Your thinking is spot on! I've implemented two layers here: on the Agent side, reporting is based on the Span in the request attribute, not dependent on ThreadLocal, so the entry isn't lost during async dispatch; on the Server side, pickRoot prioritizes spanKind=SERVER among same-level candidates, then uses the earliest startTime, so it won't degrade to picking the first outbound CLIENT. Only when an entire trace has no SERVER spans stored will it show as a downstream interface — that's a data gap, not a problem with the root-selection logic.