Spring Insight: Tracing a Request from Gateway to Loyalty in One Waterfall
Using the
GET /order/createrequest as an example, we trace it from the browser through the gateway, across the product/user/loyalty services, and use the console'scall timelineto clarify instrumentation, propagation, and waterfall parent-child relationships.Companion repositories: spring-insight · spring-insight-sca-demo
1. Introduction
When troubleshooting distributed systems, the worst thing is: I clearly called /order/create, but observability only shows some downstream interface.
Spring Insight takes a lightweight approach: add a Starter on the business side, run an insight-server alongside, and use Span + W3C traceparent to string a request into a single Trace.
This article focuses on one real chain:
Corresponding business entry: GET /order/create?userId=1&productId=1 (accessed via the gateway).
2. Build an Overall Impression with a Diagram
2.1 What the Topology Looks Like
2.2 This Entry in the Trace List
2.3 Call Timeline (Waterfall)
3. Why Does One Request Show 13 Call Timeline Entries
First, look at what the order service does in the demo - the downstream order in the waterfall almost mirrors the execution order of this code.
sca-order: OrderController#createOrder
@GetMapping("/create")
public String createOrder(Long userId, Long productId) {
// ① Get product price
BigDecimal price = productFeignClient.getPrice(productId);
// ② Get current score
Integer currentScore = userFeignClient.getScore(userId);
// ③ Add score based on price
Integer addScore = price.intValue();
Integer finalScore = userFeignClient.addScore(userId, currentScore, addScore);
return "Order created...";
}
sca-user does not calculate the score itself, but passes it through to sca-loyalty:
@GetMapping("/score/{id}")
public Integer getScore(Long id) {
return loyaltyFeignClient.getScore(id);
}
@GetMapping("/addScore")
public Integer addScore(...) {
return loyaltyFeignClient.addScore(...);
}
So one /order/create business-wise:
Gateway forwarding
→ Order.create
→ Product.price (serial 1)
→ User.score → Loyalty.score (serial 2)
→ User.addScore → Loyalty.addScore (serial 3)
In the waterfall, if "product and user run in parallel" appears, that is an anomaly; normally there should be three Feign segments one after another.
4. Top to Bottom: Why Each Step Appears
Below, explain according to the console's "call timeline" visual order (top to bottom). Durations are from one real run (magnitudes fluctuate, structure should be stable).
Row 1: sca-gateway · GET /order/create · SERVER · ~35ms
- What: The inbound Span from the browser hitting the gateway.
- Who creates it:
ReactiveInsightWebFilteron the WebFlux side (attached to theServerWebExchangeattribute, avoiding ThreadLocal loss when the event loop switches threads). - Why it is longest: It wraps the entire proxy plus all downstream work, so duration ≈ total trace duration.
Row 2: sca-gateway · GET /order/create → sca-order · CLIENT · ~33ms
- What: The gateway's outbound proxy to the order service, a CLIENT Span.
- Who creates it:
InsightGatewayTracingFilter. The operation name is stillmethod + path, so it looks the same as the inbound; the UI distinguishes with theCLIENTbadge +→ remoteService. - Parent-child: parent = the gateway SERVER from the previous row; injects W3C
traceparentdownstream (using this CLIENT's spanId as the downstream parent).
Row 3: sca-order · GET /order/create · SERVER · ~25ms
- What: The order service's inbound Span after receiving the gateway forwarding.
- Who creates it: Servlet
HttpRequestInterceptor(preHandlepushes onto stack,afterCompletionends and reports). - Cross-service continuity: Parses
traceparentfrom the request header, sharestraceIdwith the upstream,parentSpanId= the gateway CLIENT's spanId. - Why it must be reported successfully: If the ingress SERVER is lost, the list root incorrectly falls to the earliest outbound CLIENT (e.g.,
GET sca-product/...). Since Agent0.4.1,afterCompletionends the Span based on the one stored in the request attribute and reports a snapshot, no longer relying only on ThreadLocalendSpan.
Rows 4–5: Get Product Price (Serial Segment 1)
| Row | Service | Operation (schematically) | Kind | Role |
|---|---|---|---|---|
| 4 | sca-order | GET sca-product/product/price/1 |
CLIENT | Feign outbound |
| 5 | sca-product | GET /product/price/1 |
SERVER | Product inbound |
- Corresponds to
productFeignClient.getPrice(productId). - CLIENT is created by the Starter's
TracingFeignClient: it is not pushed onto the TraceContext stack, and is reported immediately after finish; the parent Span is the current ThreadLocal stack top (the order SERVER). - Common observation: CLIENT duration (e.g., 164ms) > SERVER (e.g., 26ms) – the difference is load balancing/network/connection overhead, which is normal.
Rows 6–9: Get Score (Serial Segment 2)
| Row | Service | Operation (schematically) | Kind |
|---|---|---|---|
| 6 | sca-order | GET sca-user/user/score/1 |
CLIENT |
| 7 | sca-user | GET /user/score/1 |
SERVER |
| 8 | sca-user | GET sca-loyalty/loyalty/score/1 |
CLIENT |
| 9 | sca-loyalty | GET /loyalty/score/1 |
SERVER |
- Corresponds to
userFeignClient.getScore→loyaltyFeignClient.getScore. - On the timeline it should appear after the product segment (synchronous, not parallel).
- Indentation shows the parent-child: order CLIENT → user SERVER → user CLIENT → loyalty SERVER.
Rows 10–13: Add Score (Serial Segment 3)
| Row | Service | Operation (schematically) | Kind |
|---|---|---|---|
| 10 | sca-order | GET sca-user/user/addScore?... |
CLIENT |
| 11 | sca-user | GET /user/addScore |
SERVER |
| 12 | sca-user | GET sca-loyalty/loyalty/addScore?... |
CLIENT |
| 13 | sca-loyalty | GET /loyalty/addScore |
SERVER |
- Corresponds to
addScore = price.intValue()followed byuserFeignClient.addScore. - Query parameters like
addScore=5899are the integer part of the product price at that moment – waterfall parameters directly match business.
Structure Overview (corresponding to the waterfall)
① Gateway SERVER GET /order/create
└─② Gateway CLIENT GET /order/create → sca-order
└─③ Order SERVER GET /order/create
├─④ Order CLIENT → ⑤ Product SERVER /product/price/1
├─⑥ Order CLIENT → ⑦ User SERVER /user/score/1
│ └─⑧ User CLIENT → ⑨ Loyalty SERVER
└─⑩ Order CLIENT → ⑪ User SERVER /user/addScore
└─⑫ User CLIENT → ⑬ Loyalty SERVER
5. Implementation Breakdown: How a Span Is Born and How Parent-Child Is Attached
5.1 Role Division
| Component | Responsibility |
|---|---|
| insight-agent / starter | Inbound SERVER, outbound CLIENT, sampling, W3C inject/extract, async batch reporting |
| insight-server | Receives Spans, aggregates by traceId, chooses root for list, waterfall layout and critical path |
| Console UI | Topology / list / waterfall; presentation layer distinguishes SERVER/CLIENT (does not change operationName in storage) |
5.2 Inbound SERVER (Two Runtimes)
Servlet (order / product / user / loyalty)
HttpRequestInterceptor.preHandle:TraceContext.startSpan(method + uri, remoteTraceId, remoteParent, …), setsspanKind=SERVER, stores the Span in the request attribute.- Controller Feign / RestTemplate runs.
afterCompletion: ends the Span stored on the request and reports asnapshot, thenTraceContext.clear().
Key point: outbound CLIENT is not pushed onto the stack; the stack should always hold the current HTTP SERVER. If async dispatch empties ThreadLocal, the request attribute still allows the ingress span to report - this is the core fix in 0.4.1.
WebFlux / Gateway
- Inbound:
ReactiveInsightWebFilterattaches the SERVER Span to the exchange attribute + Reactor Context. - Outbound:
InsightGatewayTracingFilterreads the inbound SERVER as parent, creates the CLIENT, and rewrites outbound request headers.
5.3 Outbound CLIENT (Feign / RestTemplate / Gateway)
Common:
parent = TraceContext.currentSpan()(or the exchange's SERVER in Gateway).new TraceSpan(parent.traceId, parent.spanId)→ this CLIENT's spanId becomes the parent for the downstreamtraceparent.spanKind=CLIENT,remoteServiceuses a logical service name when possible (lb:/// Feign Target / non-IP Host), to avoid topology edges becoming127.0.0.1.- After the call,
finish+reportSpan(snapshot), not pushed onto the TraceContext stack.
This shapes the classic "CLIENT wraps downstream SERVER" in the waterfall.
5.4 W3C traceparent: The Glue Across Processes
When spring.insight.http-trace-propagation-enabled (default true) is on:
traceparent: 00-<trace-id>-<parent-span-id>-<flags>
- Outbound: writes the CLIENT spanId into the parent field.
- Inbound: on new SERVER,
traceIdis inherited,parentSpanId= the peer CLIENT. - Sampling flags propagate; if not sampled, no reporting.
Without propagation, each service would start its own Trace, and the topology and waterfall would break.
5.5 How the Server Draws the Waterfall
- Fetch all Spans by
traceId. - Build a tree using
parentSpanIdto compute indentation depth; those without a parent (cross-service entry) are treated as "local root". - Sort by
startTime, draw bars by relative offset. - List "root operation name" priority: no parent → parent not in this Trace → earliest start; when tied, prefer
SERVER. - UI adds badges and
→ remoteServiceto solve the hard-to-distinguish Gateway SERVER/CLIENT with the same name.
6. Two Pitfalls (Encountered in This Chain)
Pitfall A: List Root Becomes sca-product/...
Phenomenon: You accessed /order/create, but the most recent request shows the product CLIENT.
Cause: The order's ingress SERVER was not stored, so the list picks the earliest Span among those "parent not in this Trace" - which happens to be the first Feign CLIENT.
Fix direction: The inbound completion path must always report the SERVER attached to the request/exchange (Agent 0.4.1).
Pitfall B: The First Two Waterfall Rows Both Show GET /order/create
Phenomenon: Looks like a duplicate request.
Cause: The gateway's inbound SERVER and outbound CLIENT intentionally share the same path name (the real HTTP path is this).
Display optimization: badge distinguishes Kind; CLIENT appends → sca-order (UI only, not changing reported fields).
7. Reproduce This Chain Yourself
- Start insight-server (e.g.,
ghcr.io/iweidujiang/spring-insight-server). - Start spring-insight-sca-demo https://github.com/iweidujiang/spring-insight-sca-demo (contains gateway / order / product / user / loyalty).
- Use Agent ≥ 0.4.1 (includes the ingress SERVER reporting fix).
- Browser access:
http://localhost:8080/order/create?userId=1&productId=1(actual port per demo). - Open console → Trace → click that Trace, compare with Section 4.
8. Summary
- What decides the waterfall order: the order of synchronous business calls (in the demo, the three Feign segments).
- Who decides parent-child? In-process: stack top; cross-process:
traceparentparent = upstream CLIENT. - Why does the gateway have two rows with the same name? SERVER (inbound) + CLIENT (forward), same path; they are distinguished by Kind /
→ downstream. - Why must the list root be the ingress? The root operation should align with the user's perspective; if the ingress SERVER is lost, it gets "impersonated by a downstream CLIENT".
Spring Insight deliberately does not build a second full APM: making HTTP inbound/outbound + W3C + one adequate waterfall solid is enough to explain "the order chain".
If JDBC/MQ are added later, they will extend the same "CLIENT attaches to SERVER, propagation attaches across processes" model.
Finally, the repository is here:
https://github.com/iweidujiang/spring-insight
Please give a Star, file Issues.
Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.
Lost entries can cause the root node to be misjudged.
Your thinking is spot on! I've implemented two layers here: on the Agent side, reporting is based on the Span in the request attribute, not dependent on ThreadLocal, so the entry isn't lost during async dispatch; on the Server side, pickRoot prioritizes spanKind=SERVER among same-level candidates, then uses the earliest startTime, so it won't degrade to picking the first outbound CLIENT. Only when an entire trace has no SERVER spans stored will it show as a downstream interface — that's a data gap, not a problem with the root-selection logic.