跪拜 Guibai
← All articles
Backend · Spring · Open Source

Spring Insight: Tracing a Request from Gateway to Loyalty in One Waterfall

By 苏渡苇 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Distributed tracing often feels like a black box; this open-source implementation spells out the exact span lifecycle and propagation rules, so engineers can reason about their own traces. The included demo reproduces the full chain locally, and the explained pitfalls map to real debugging scenarios.

Summary

Spring Insight pairs a business-side starter with a standalone insight-server to assemble spans into traces. The project deliberately avoids a full APM suite, focusing on HTTP ingress/egress plus W3C traceparent propagation. A demo with gateway, order, product, user, and loyalty services shows what a real request produces: 13 waterfall entries that mirror the synchronous Feign calls in the controller code. Each span's creation is explained, from the ReactiveInsightWebFilter at the gateway to the Servlet HttpRequestInterceptor in downstream services. The article also walks through two pitfalls: the list root becoming a client span when the ingress server span is lost, and two gateway rows sharing the same path where a CLIENT badge plus arrow disambiguates them. Version 0.4.1 fixes the root problem by always reporting the request-attached SERVER span, even when ThreadLocal empties in async dispatch.

Takeaways
— A single /order/create request produces 13 waterfall entries: one gateway SERVER, one gateway CLIENT, one order SERVER, then three serial Feign segments (product, score, addScore) each containing a CLIENT and SERVER pair.
— Waterfall order mirrors the synchronous execution order of Feign calls; parallel product and user calls would indicate an anomaly.
— CLIENT spans are never pushed onto the trace context stack; they finish and report immediately after the call, so the current ThreadLocal top is always the HTTP SERVER.
— W3C traceparent uses the CLIENT spanId as the parent for the downstream SERVER, connecting traces across process boundaries.
— Version 0.4.1 fixes a missing ingress SERVER by ending the span from the request attribute, not just ThreadLocal, ensuring the list root aligns with the user-facing entry.
— A CLIENT span can take longer than its downstream SERVER (e.g., 164ms vs 26ms) due to load balancing, network, or connection overhead, which is normal.
Conclusions

Keeping CLIENT spans off the stack simplifies thread-local management but means the ingress SERVER span must be attached to the request or exchange for reliable reporting, a nontrivial design choice.

The decision to deliberately not build a full APM keeps the project focused and its debugging story understandable; extending to JDBC or MQ would reuse the same parent-child model.

The two pitfalls highlight how easy it is to misattribute a trace root when entry spans are dropped, a lesson applicable to any tracing system.

Concepts & terms
Span
A unit of work in a trace, recording a single operation with a start time, duration, and parent relationship.
W3C traceparent
An HTTP header that carries trace ID, parent span ID, and flags, enabling distributed traces to link across service boundaries.
Feign
A declarative HTTP client for Spring Boot that simplifies calling REST services; here it generates the client spans.
From the discussion
Featured comments
卷土的土

Lost entries can cause the root node to be misjudged.

苏渡苇

Your thinking is spot on! I've implemented two layers here: on the Agent side, reporting is based on the Span in the request attribute, not dependent on ThreadLocal, so the entry isn't lost during async dispatch; on the Server side, pickRoot prioritizes spanKind=SERVER among same-level candidates, then uses the earliest startTime, so it won't degrade to picking the first outbound CLIENT. Only when an entire trace has no SERVER spans stored will it show as a downstream interface — that's a data gap, not a problem with the root-selection logic.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗