Spring Insight: Tracing a Request from Gateway to Loyalty in One Waterfall
Distributed tracing often feels like a black box; this open-source implementation spells out the exact span lifecycle and propagation rules, so engineers can reason about their own traces. The included demo reproduces the full chain locally, and the explained pitfalls map to real debugging scenarios.
Spring Insight pairs a business-side starter with a standalone insight-server to assemble spans into traces. The project deliberately avoids a full APM suite, focusing on HTTP ingress/egress plus W3C traceparent propagation. A demo with gateway, order, product, user, and loyalty services shows what a real request produces: 13 waterfall entries that mirror the synchronous Feign calls in the controller code. Each span's creation is explained, from the ReactiveInsightWebFilter at the gateway to the Servlet HttpRequestInterceptor in downstream services. The article also walks through two pitfalls: the list root becoming a client span when the ingress server span is lost, and two gateway rows sharing the same path where a CLIENT badge plus arrow disambiguates them. Version 0.4.1 fixes the root problem by always reporting the request-attached SERVER span, even when ThreadLocal empties in async dispatch.
Keeping CLIENT spans off the stack simplifies thread-local management but means the ingress SERVER span must be attached to the request or exchange for reliable reporting, a nontrivial design choice.
The decision to deliberately not build a full APM keeps the project focused and its debugging story understandable; extending to JDBC or MQ would reuse the same parent-child model.
The two pitfalls highlight how easy it is to misattribute a trace root when entry spans are dropped, a lesson applicable to any tracing system.
Lost entries can cause the root node to be misjudged.
Your thinking is spot on! I've implemented two layers here: on the Agent side, reporting is based on the Span in the request attribute, not dependent on ThreadLocal, so the entry isn't lost during async dispatch; on the Server side, pickRoot prioritizes spanKind=SERVER among same-level candidates, then uses the earliest startTime, so it won't degrade to picking the first outbound CLIENT. Only when an entire trace has no SERVER spans stored will it show as a downstream interface — that's a data gap, not a problem with the root-selection logic.