SkyWalking Ships a Three-Layer AI Stack That Reads Traces So You Don't Have To
Foreword
The "last mile" of distributed tracing has finally been bridged by AI.
Recently, while troubleshooting online issues, I noticed a very common phenomenon — everyone uses SkyWalking to view traces, but very few can truly "understand" them.
Tracing tools draw out the distributed call chain, but the core steps of troubleshooting — "finding the slow one among a pile of Spans," "locating the root cause from alerts," "summarizing patterns from massive Traces" — all rely on manual work.
A colleague showed me their team's troubleshooting workflow: an alert comes in, they glance at SkyWalking's topology map and see a red line;
switch to the Trace list, filter services, filter interfaces, flip pages, and find a few slow ones;
open the Span tree, read it themselves, compare it themselves, guess themselves;
then switch to the log system to match TraceIDs, and then switch to the monitoring system to check metrics.
The whole process takes at least 20 minutes, sometimes one or two hours.
He said something that stuck with me: "SkyWalking draws the call chain, but the 'last mile' of understanding it and using it to solve problems still relies on people."
Now, this problem finally has a solution.
Apache SkyWalking has officially integrated AI capabilities!
This is not a concept, but a complete set of observable AI capability matrices that have already landed and are rapidly evolving.
From the AI intelligent Q&A assistant in Horizon UI, to Virtual GenAI's large model application monitoring, to AI Pipeline's machine learning anomaly detection — SkyWalking is evolving from a "trace display tool" into an "intelligent diagnostic platform."
This article will discuss this topic specifically with everyone, hoping it will be helpful to you.
More project practices on my technical website: susan.net.cn/project
1. What are the integrated AI capabilities?
Before diving deep, let's establish an overall understanding.
SkyWalking's AI capabilities are not as simple as "adding a chat box"; they cover three different layers:
Layer 1: Horizon UI AI Assistant — Ask your observation data in natural language
This is the core feature of Horizon UI 1.0, just released in July 2026.
Simply put, you no longer need to click around a bunch of dashboards to find data — just ask the AI directly.
You can directly ask: "What unhealthy things are currently in the system?" or "Help me check why the response time of a certain service has slowed down".
The AI reads real-time data from the OAP backend and then generates a readable report with real charts for you.
It doesn't "translate numbers on the dashboard into text," but truly understands your observation data and then explains it clearly to you.
Layer 2: Virtual GenAI — Make your large model calls no longer a "black box"
If your application uses Spring AI or OpenAI SDK to call large models, SkyWalking 10.4's Virtual GenAI lets you see: the Latency of each LLM call, Token usage (input/output), estimated cost, and the Time to First Token (TTFT) for streaming calls.
"Which model is your application calling, how much does it cost, and is the user experience fast" — previously these were all black boxes, now SkyWalking visualizes them all for you.
Layer 3: AI Pipeline — Machine learning makes operations "proactive"
SkyWalking's AI Pipeline module communicates with third-party AI services via the gRPC protocol, providing two core capabilities: intelligent baseline calculation and HTTP URI pattern recognition.
It uses machine learning algorithms to automatically learn the normal patterns of application behavior, immediately alerting when abnormal deviations are detected, without the need for manual threshold configuration.
At the same time, SkyWalking also supports intelligent URI pattern recognition for RESTful APIs, automatically aggregating and classifying a large number of HTTP requests.
Adding these three layers of capabilities together, SkyWalking is transforming from a "trace display tool" into a true "intelligent observability platform."
2. Horizon UI AI Assistant
It can replace mouse clicks with conversation.
2.1 What exactly is it?
Horizon UI is SkyWalking's next-generation web console.
And the AI Assistant is an AI intelligent assistant built into Horizon UI 1.0.
Its core design philosophy can be summed up in one sentence: "Show, don't describe."
What does that mean?
Typical AI Q&A gives you a lot of text.
The AI Assistant is different — it writes a one or two sentence summary, then directly draws a real chart, and then explains what this chart indicates.
Each chart is a real, real-time rendering, not a screenshot, nor fabricated numbers.
2.2 What can it see?
The AI Assistant answers questions through three data sources, which must be combined to piece scattered signals into a complete picture:
① Real-time observation data It uses the exact same OAP query protocol as the dashboard, so the data it sees is exactly the same as what you see on your dashboard.
② Layer configuration (as a skill library) The Layer configuration acts like the AI Assistant's "knowledge directory," telling it what metrics each Layer measures and the entity scope of each metric.
③ Permission system It is read-only and inherits your permissions. You can only see data you have permission to view.
2.3 It is disabled by default
The AI Assistant is disabled by default and needs to be manually enabled by operations personnel, pointing to a model service.
You can use your own model (either a local Ollama or a cloud API), costs are controllable, and data does not leave the country.
3. Virtual GenAI
It is the "digital dashboard" for LLM calls.
3.1 Pain Point: Applications are "consuming" large models, but monitoring has left a blind spot
After many development teams quickly integrated large model capabilities using Spring AI or OpenAI SDK, they discovered an awkward problem — they knew almost nothing about the actual performance of these calls.
You only know you pay a certain provider a large sum each month, but what is the "input-output ratio" of this bill within the application?
Spending several times the Token cost for a flagship model, is it really faster in your actual business scenario?
Wanting to set a timeout for LLM calls, should it be 30 seconds or 60 seconds?
The result of a gut-feel decision is often setting it too short, causing requests to be frequently interrupted, or setting it too long, causing request pile-ups that drag down the entire Java application.
The Virtual GenAI capability introduced in SkyWalking 10.4 is precisely to solve this "observation vacuum."
3.2 Technical Principle
When the SkyWalking Java Agent or OTLP probe intercepts calls to mainstream AI frameworks (such as Spring AI, OpenAI SDK), they report Trace data to the SkyWalking OAP.
This Trace data follows the GenAI semantic conventions, containing key information such as Provider name, model name, input/output Token counts, and Time to First Token (TTFT).
The OAP extracts these metrics from the distributed tracing data and displays traffic, latency, success rate, Token usage (input/output), and estimated cost on the GenAI dashboard.
3.3 What can you see?
After integrating Virtual GenAI, you can see detailed metrics at the Provider level and Model level on the GenAI dashboard:
| Metric Category | Specific Metric | Description |
|---|---|---|
| Traffic & Success Rate | CPM (Calls Per Minute), SLA (Success Rate) | Call volume and health |
| Latency | P50/P90/P99 Latency Percentiles | Real fluctuation curve, providing a basis for dynamic timeouts |
| TTFT (Time to First Token) | First token latency for streaming calls | Core UX metric |
| Token Usage | Input Tokens / Output Tokens | Usage statistics |
| Estimated Cost | Estimation based on configured unit costs | Every bill is clearly calculated |
Configuring Cost Estimation: Configure the pricing of Providers and Models in gen-ai-config.yml, and SkyWalking will automatically calculate the estimated cost for each call.
3.4 Two Monitoring Modes
SkyWalking supports two large model monitoring modes:
Mode 1: Gateway-side Monitoring (Envoy AI Gateway) Unify LLM traffic access through Envoy AI Gateway, which pushes GenAI metrics and access logs directly to SkyWalking OAP via OTLP gRPC. Suitable for scenarios managing multiple Provider calls uniformly.
Mode 2: Application-side Monitoring (Virtual GenAI) Collect the most authentic data from the client perspective through the Java Agent. Suitable for monitoring down to every single SDK call.
4. AI Pipeline
It is machine learning-driven intelligent operations.
If the Horizon UI AI Assistant solves the problem of "how to query," and Virtual GenAI solves the problem of "how to monitor LLMs," then AI Pipeline solves the problem of "how to make operations proactive."
4.1 Architectural Principle
SkyWalking's AI Pipeline module uses the gRPC protocol to communicate with third-party AI services, supporting asynchronous processing of large-scale telemetry data.
The core includes the following components:
- Baseline Query Service: Supports intelligent baseline calculation for time-series data
- HTTP URI Recognition Service: Implements intelligent recognition and classification of URL patterns
- Prediction Service Metrics: Provides machine learning model performance monitoring and evaluation capabilities
4.2 Four Core AI Functions
① Intelligent Anomaly Detection SkyWalking uses machine learning algorithms to automatically learn the normal patterns of application behavior, immediately alerting when abnormal deviations are detected, without the need for manual threshold configuration.
② Accelerated Root Cause Analysis By correlating and analyzing multi-dimensional metrics, the AI engine can quickly locate the root cause of problems, significantly reducing Mean Time to Repair.
③ Adaptive Baseline Calculation The system can automatically adapt to the cyclical changes of business traffic, distinguishing true anomalies from normal business fluctuations.
④ Intelligent URI Pattern Recognition Automatically identifies and classifies HTTP request URL patterns, helping operations personnel understand API usage. Implemented based on the R3 algorithm (RESTful Pattern Recognition), it uses machine learning to perform pattern recognition and grouping on RESTful APIs.
4.3 Deployment Configuration
Enable the AI Pipeline module and configure the gRPC endpoint in application.yml:
ai-pipeline:
enabled: true
grpc-endpoint: "your-ai-service:50051"
The system supports capabilities such as TensorFlow/PyTorch model integration, real-time streaming data processing, distributed model inference, and model performance monitoring.
5. How to Integrate SkyWalking AI?
5.1 Integrating Horizon UI AI Assistant
Step 1: Deploy Horizon UI
Horizon UI is released as an independent container image, available from Docker Hub:
docker pull apache/skywalking-ui:horizon-latest
docker run -d -p 8080:8080 \
-e OAP_BACKEND_URL=http://your-oap:12800 \
apache/skywalking-ui:horizon-latest
Step 2: Enable AI Assistant
Enable the AI Assistant in Horizon UI's configuration file and configure the model service endpoint:
ai:
assistant:
enabled: true
model:
endpoint: "http://your-llm-service:8000/v1"
api_key: "your-api-key" # Optional
Supports various model services such as Ollama local models, OpenAI API, Gemini API, etc.
Step 3: Start Asking Questions
After logging into Horizon UI, click the AI Assistant icon and directly input natural language questions, for example:
- "What unhealthy services are currently in the system?"
- "Why has order-service slowed down recently?"
- "Help me check the P99 latency trend for the payment interface"
5.2 Integrating Virtual GenAI (Java Application)
Step 1: Upgrade Agent Version
Ensure the SkyWalking Java Agent version >= 9.7.
Step 2: Add Dependency in pom.xml
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>apm-toolkit-trace</artifactId>
<version>9.7.0</version>
</dependency>
Step 3: Configure Cost Estimation
Configure model pricing in SkyWalking OAP's config/gen-ai-config.yml:
providers:
- provider: openai
prefix-match:
- gpt
models:
- name: gpt-4o
input-estimated-cost-per-m: 2.5
output-estimated-cost-per-m: 10
Step 4: Start Application and View GenAI Dashboard
After starting the application, find the GenAI Dashboard in Horizon UI to see the traffic, latency, Token usage, and estimated cost of all LLM calls.
5.3 Integrating AI Pipeline
Step 1: Deploy AI Service
Deploy a service that implements the SkyWalking AI Pipeline gRPC protocol (open-source R3 algorithm implementations can be used).
Step 2: Configure OAP
Enable AI Pipeline in OAP's application.yml:
ai-pipeline:
enabled: true
grpc-endpoint: "your-ai-service:50051"
Step 3: Enable URI Pattern Recognition
Enable HTTP URI pattern recognition in the OAP configuration, and the AI will automatically perform pattern recognition and grouping on RESTful APIs.
6. Advantages and Disadvantages
Advantages
1. Natural language intelligent Q&A significantly improves troubleshooting efficiency The AI Assistant transforms troubleshooting from "flipping through lists, reading Span trees, writing fault groups" to "ask a question → get a traceId → verify with a chart." A troubleshooting process that originally took 20 minutes can potentially be compressed to under 5 minutes.
2. LLM calls become observable, every penny is accounted for Virtual GenAI turns LLM calls from a "black box" into a "white box" — key metrics like Token usage, estimated cost, and TTFT are clear at a glance.
3. Machine learning intelligent operations, say goodbye to manual threshold configuration AI Pipeline automatically learns application behavior patterns, with intelligent alerting and root cause analysis requiring no manual intervention.
4. Layered integration, progressive implementation AI capabilities are integrated independently across three layers; you can start with the one you need most without having to adopt everything at once.
5. Flexible models, controllable costs The AI Assistant supports connecting to local Ollama or cloud APIs, data stays within your borders, and costs are under your control.
6. Open-source ecosystem continuously evolving As an Apache top-level project with a 9-year history and thousands of cluster deployments verified, SkyWalking's AI capabilities are still rapidly iterating.
Disadvantages and Considerations
1. AI Assistant is disabled by default and requires manual configuration The AI Assistant is disabled by default and requires operations personnel to manually enable and configure the model service.
2. Virtual GenAI requires an Agent version upgrade Requires SkyWalking Java Agent >= 9.7; older versions need to be upgraded.
3. AI Pipeline requires a third-party AI service The AI Pipeline itself does not contain an AI model and requires deploying an additional AI service (such as an R3 algorithm implementation).
4. Cost estimation requires manual configuration
Virtual GenAI's cost estimation requires manually configuring the pricing of each model in gen-ai-config.yml.
5. AI capabilities are still rapidly evolving Some AI features (such as natural language to BydbQL) are still in the development or preview stage.
7. Applicable Scenarios
| Scenario | Recommendation Level | Recommended Capability | Reason |
|---|---|---|---|
| Daily Troubleshooting/On-Call | ✅✅✅ Highly Recommended | AI Assistant | Transforms "flipping through logs" into "asking AI" |
| Large Model Application Development | ✅✅✅ Highly Recommended | Virtual GenAI | Monitor the performance and cost of LLM calls |
| Multi-Provider LLM Management | ✅✅✅ Highly Recommended | Virtual GenAI + Envoy AI Gateway | Unified observation of Tokens and costs across multiple Providers |
| Large-scale Microservice Operations | ✅✅ Recommended | AI Pipeline | Intelligent anomaly detection + root cause analysis |
| E-commerce Promotion Capacity Planning | ✅✅ Recommended | AI Pipeline | Predict traffic peaks, warn of bottlenecks in advance |
| Resource Utilization Optimization | ✅✅ Recommended | AI Pipeline | AI recommends optimal resource allocation strategies |
More project practices on my technical website: susan.net.cn/project
Summary
Returning to the initial question: What problem does SkyWalking's integration of AI actually solve?
It solves the "last mile" problem of tracing tools.
Previously, SkyWalking was like an X-ray machine — capable of taking clear skeletal images, but understanding the film and making a diagnosis still relied on the doctor.
Now, SkyWalking + AI is like an AI-assisted diagnostic system — not only does it take the image, but it can also automatically identify lesions, give diagnostic suggestions, and answer your questions.
Horizon UI AI Assistant lets you ask observation data in natural language; Virtual GenAI turns LLM calls from a black box into a white box; AI Pipeline transforms operations from passive response to proactive prediction.
With these three layers of capabilities superimposed, SkyWalking is evolving from a "trace display tool" into an intelligent observability platform that truly helps developers and operations personnel "understand the system and quickly locate problems."
If you are still troubleshooting by "flipping through Trace lists, reading Span trees, and manually correlating logs," it is recommended you spend an afternoon trying to run Horizon UI + AI Assistant.
Experience the troubleshooting experience of "getting an answer with just one question" — you will find that what used to take 20 minutes might now only take 3 minutes.