跪拜 Guibai
← Back to the summary

How Distributed Tracing Rebuilds Visibility in Microservices

In a microservice architecture, a single request often involves multiple modules, multiple middleware components, and collaboration across multiple machines. Some of these calls are serial, while others are parallel. So how do you determine which applications, modules, and nodes are invoked behind a request, and in what order? How do you pinpoint the performance bottlenecks of each module? This article will reveal the answers.

This article will elaborate on the following aspects

Principles and Functions of Distributed Tracing Systems

How do you measure the performance of an interface? Generally, we focus on at least the following three metrics:

Monolithic Architecture

In the early stages, when a company is just starting, it often adopts a monolithic architecture like the one below. How should we calculate the three metrics above for a monolithic architecture?

The most obvious approach is to use AOP.

By using AOP to print the time before and after calling specific business logic, you can calculate the overall call time. Using AOP to catch exceptions also reveals where the call error occurred.

Microservice Architecture

In a monolithic architecture, all services and components are on a single machine, making these monitoring metrics relatively easy to implement. However, with rapid business growth, a monolithic architecture inevitably evolves into a microservice architecture, as shown below.

As illustrated: a somewhat complex microservice architecture.

If a user reports that a certain page is very slow, and we know the request call chain for this page is A -----> C -----> B -----> D, how do we pinpoint which module might be causing the problem? Each service (A, B, C, D) has several machines. How do we know which specific machine a request called?

It's clear that because we cannot accurately locate the exact path of each request, the following pain points exist in a microservice architecture:

  1. Troubleshooting is difficult and time-consuming.
  2. Specific scenarios are hard to reproduce.
  3. Analyzing system performance bottlenecks is challenging.

Distributed call chains were created to solve these problems. Their main functions are:

A distributed tracing system can effectively locate each specific request link as shown below, making it easy to implement request link tracing and analyze the performance bottlenecks of each module.

Distributed Call Chain Standard - OpenTracing

Knowing the purpose of distributed call chains, let's look at how they are implemented and their principles. First, to solve the problem of incompatible APIs among different distributed tracing systems, the OpenTracing specification was born. OpenTracing is a lightweight standardization layer that sits between applications/class libraries and tracing or log analysis programs.

Speaking of this, have you ever thought about a similar implementation in Java? Remember JDBC? By providing a set of standard interfaces for various vendors to implement, programmers can program to interfaces without worrying about the specific implementation. The interface here is essentially the standard, so establishing a standard is very important for achieving component pluggability.

Next, let's look at the OpenTracing data model, which mainly consists of the following three concepts:

Understanding these three concepts is very important. To help you better understand them, I have drawn a diagram.

As shown, a complete request for placing an order is a Trace. Obviously, for this request, there must be a global identifier to mark it. Each call is called a Span, and each call must carry the global TraceId so that the global TraceId can be associated with each call. This TraceId is transmitted through SpanContext. Since it needs to be transmitted, calls must obviously follow a protocol. As illustrated, it might be easier to understand if we compare the transmission protocol to a car, SpanContext to cargo, and Span to a road.

Having understood these three concepts, let's look at how a distributed tracing system collects the microservice call chain in the unified diagram.

We can see a Collector at the bottom layer that is constantly collecting data. What information does the Collector collect for each call?

  1. Global trace_id: This is obvious, as it's needed to associate each sub-call with the original request.
  2. span_id: 0, 1, 1.1, 2 in the diagram, which identifies which call it is.
  3. parent_span_id: For example, if the span_id for b calling d is 1.1, then its parent_span_id is the span_id for a calling b, which is 1. This is how two adjacent calls are associated.

With this information, the information collected by the Collector for each call is as follows:

Based on this chart information, a visual view of the call chain can obviously be drawn as follows:

Thus, a complete distributed tracing system is realized.

The above implementation seems simple, but there are several issues that require careful consideration:

  1. How to automatically collect span data: Automatic collection, with no intrusion into business code.
  2. How to pass context across processes.
  3. How to ensure the traceId is globally unique.
  4. With such a high request volume, will collection affect performance?

Next, let's see how SkyWalking solves these four problems.

SkyWalking's Principles and Architecture Design

How to Automatically Collect Span Data

SkyWalking uses a plugin + javaagent approach to achieve automatic span data collection. This ensures no intrusion into the code. The plugin approach means it's pluggable and has good extensibility (later we will discuss how to define your own plugins).

How to Pass Context Across Processes

We know that data is generally divided into header and body, just like HTTP has a header and body, and RocketMQ has MessageHeader and Message Body. The body usually contains business data, so it's not appropriate to pass context in the body; context should be passed in the header, as shown.

How to Ensure the traceId is Globally Unique

To ensure global uniqueness, we can use a distributed or locally generated ID. Using a distributed approach requires an ID generator, and each request would need to request this generator first, incurring a network call overhead. Therefore, SkyWalking ultimately adopted a local ID generation method, using the famous snowflake algorithm, which offers high performance.

Diagram: ID generated by the snowflake algorithm

However, the snowflake algorithm has a well-known problem: clock rollback, which can lead to duplicate IDs. So how does SkyWalking solve the clock rollback problem?

Here we need to discuss the trade-offs in system design. First, if we were to perform a uniqueness check on the generated random number, it would undoubtedly add another layer of calls and cause some performance loss. However, the probability of clock rollback occurring is very small (when it does happen, the business is significantly affected due to machine time disorder, so adjusting machine time must be done with extreme caution). Coupled with the very low probability of generating duplicate random numbers, there is really no need to add another layer of global uniqueness verification here. When selecting technical solutions, one must avoid over-engineering; going too far is as bad as not going far enough.

With Such a High Request Volume, Will Collecting Everything Affect Performance?

If we collect data for every request call, the data volume will undoubtedly be massive. But conversely, is it really necessary to collect data for every request? It's not necessary; we can set a sampling frequency and only sample a portion of the data. SkyWalking defaults to sampling 3 times every 3 seconds, with the remaining requests not being sampled, as shown.

This would lead to a situation where some calls are sampled on Service A but not on Services B and C, making it impossible to analyze the performance of the call chain. So how does SkyWalking solve this?

It solves it like this: If the upstream carries a Context (indicating the upstream was sampled), then the downstream is forced to collect data. This ensures the integrity of the link.

SkyWalking's Basic Architecture

SkyWalking's basic architecture is as follows. It can be said that almost all distributed tracing systems are composed of the following components:

How is SkyWalking's Performance?

Next, you are probably concerned about SkyWalking's performance. Let's look at the official benchmark data.

In the diagram, blue represents performance without SkyWalking, and orange represents performance with SkyWalking. The data was measured at a TPS of 5000. It can be seen that whether it's CPU, memory, or response time, the performance overhead introduced by SkyWalking is almost negligible.

Next, let's look at a comparison between SkyWalking and other well-known distributed tracing tools in the industry, Zipkin and Pinpoint (compared under conditions of 1 sample per second, 500 threads, and a total of 5000 requests). It can be seen that in terms of critical response time, Zipkin (117ms) and PinPoint (201ms) are far inferior to SkyWalking (22ms)!

In terms of the performance overhead metric, SkyWalking wins hands down!

Let's look at another metric: code intrusiveness. Zipkin requires instrumentation within the application, leading to strong code intrusion. SkyWalking, by using javaagent + plugins to modify bytecode, achieves zero code intrusion. Besides its excellent performance and non-intrusive nature, SkyWalking has several other advantages:

Our Company's Practices with Distributed Call Chains

SkyWalking's Application Architecture in Our Company

From the above, we know SkyWalking has many advantages. But did we use all of its components? Not exactly. Let's look at its application architecture in our company.

As seen in the diagram, we only adopted SkyWalking's agent for sampling and abandoned the other three major components: 'Data Reporting and Analysis', 'Data Storage', and 'Data Visualization'. Why didn't we directly adopt SkyWalking's entire solution? Because before integrating SkyWalking, our Marvin monitoring ecosystem was already relatively mature. Replacing it entirely with SkyWalking was unnecessary for several reasons: first, Marvin met our needs in most scenarios; second, the system replacement cost was high; third, re-onboarding users would incur a high learning cost.

This also gives us a revelation: for any product, seizing the initiative is very important. The replacement cost for subsequent products is high. Seizing the initiative means capturing the user's mindshare. This is like WeChat, which, despite its sophisticated UI and features, still couldn't beat WhatsApp abroad because the first-mover advantage was already lost.

On the other hand, for architecture, there is no best, only the most suitable. Balancing and compromising based on the current business scenario is the essence of architecture design.

What Modifications and Practices Did Our Company Implement for SkyWalking?

Our company mainly implemented the following modifications and practices:

  1. Forced sampling in the staging environment for debugging needs.
  2. Implementing more fine-grained sampling.
  3. Embedding traceId in logs.
  4. Self-developed SkyWalking plugins.

Forced Sampling in the Staging Environment for Debugging Needs

From the analysis above, we know the Collector samples periodically in the background. Isn't that good enough? Why implement forced sampling? It's still for troubleshooting and locating problems. Sometimes, when an issue occurs online, we hope to reproduce it in the staging environment and see the complete call chain for that request. Therefore, implementing forced sampling in staging is very necessary. So, we modified SkyWalking's dubbo plugin to implement forced sampling.

We attach a key-value pair like force_flag = true to the request's Cookie to indicate we want forced sampling. After the gateway receives this Cookie, it carries the force_flag = true key-value pair in the dubbo attachment. Then, SkyWalking's dubbo plugin can determine whether it's a forced sample based on this. If this value exists, it forces sampling; otherwise, it proceeds with the normal periodic sampling.

Implementing More Fine-Grained Sampling

What is more fine-grained sampling? Let's first look at SkyWalking's default sampling method, which is uniform sampling.

How to Embed traceId in Logs?

Embedding traceId in output logs helps us troubleshoot problems, so printing the traceId is very necessary. How do you embed traceId in logs? We use log4j. Here, we need to understand log4j's plugin mechanism. Log4j allows us to customize plugins to output log formats. First, we need to define the log format, embedding %traceId as a placeholder in the custom log format, as shown below.

Then we implement a log4j plugin, as follows:

Which SkyWalking Plugins Did Our Company Self-Develop?

SkyWalking has implemented many plugins, but it did not provide plugins for memcached and druid. So, we self-developed plugins for these two according to its specifications.

How is a plugin implemented? As you can see, it mainly consists of three parts:

  1. Plugin Definition Class: Specifies the plugin's definition class, which will ultimately be used to package and generate the plugin.
  2. Instrumentation: Specifies the aspect and pointcut, i.e., which class's which method to enhance.
  3. Interceptor: Specifies the enhancement logic to be written before, after, or during an exception in the method from step 2.

You might still be confused, so let's briefly explain using the dubbo plugin. We know that in a dubbo service, each request goes through a dozen Filter processes from the moment netty receives the message and hands it over to the business thread pool, until it actually calls the business method.

So, obviously, we need to specify the class we want to enhance in the plugin (MonitorFilter) and enhance its method (invoke). What enhancements should be made to this method? This is what the Interceptor does. Let's look at the instrumentation in the Dubbo plugin (DubboInstrumentation).

Let's look at what the Interceptor described in the code does. The key steps are listed below.

First, beforeMethod means the method here will be called before executing MonitorFilter's invoke method. Correspondingly, afterMethod represents the enhancement logic after executing the invoke method.

Secondly, from points 2 and 3, we can see that whether it's a consumer or a provider, the global ID is processed accordingly. This ensures that by the time it reaches the actual business layer, this global traceid is guaranteed to exist. After defining the Instrumentation and Interceptor, the final step is to specify the defined class in the skywalking.def file.

// skywalking-plugin.def file
dubbo=org.apache.skywalking.apm.plugin.asf.dubbo.DubboInstrumentation

In this way, the packaged plugin will enhance the invoke method of MonitorFilter. Before the invoke method is executed, it performs operations like injecting the global traceId into its attachment. All of this is done silently and is non-intrusive to the code.