跪拜 Guibai
← Back to the summary

Two Years In, We Pulled Java Virtual Threads from Production

When your Spring Boot service has been running on JDK 21 for half a year, P99 latency doesn't drop but rises, thread pools mysteriously freeze, and Arthas can no longer see familiar thread names... you'll make the same decision.

In September 2024, one year after JDK 21 was released, we decided to upgrade all core services to JDK 21. The number one feature we were eyeing was Virtual Threads — at the time, almost all tech media were saying "this is a revolution in Java concurrent programming."

By March 2026, after 6 months of canary testing and production operation, we switched two of our core services back from virtual threads to platform threads.

This article is not meant to deny virtual threads — they are indeed a good thing. What I want to share are the pitfalls we encountered in a real production environment, and the final conclusion we reached: Virtual threads are not a silver bullet. Used in the right scenario, they are a sharp tool; used in the wrong scenario, they are a disaster.

Background

First, let me describe our scenario:

In the first week after the upgrade, all metrics on the monitoring dashboard looked great: throughput increased by 30%, and P99 latency dropped by about 40%. We excitedly wrote in our weekly report: "Virtual thread migration completed successfully, performance significantly improved."

Then the problems began to surface.

Pitfall 1: Pinning caused by synchronized

Phenomenon

Two weeks into operation, we started receiving sporadic timeout alerts. The P99 latency of a certain endpoint soared from 50ms to over 5s, lasting for tens of seconds before recovering automatically.

Investigation

Checking the thread dump, we found a large number of virtual threads were "pinned" to carrier threads, causing the carrier thread pool to be exhausted.

Cause

A core limitation of virtual threads: When a virtual thread enters a synchronized block or executes a native method, it gets pinned to the carrier thread, blocking the scheduling of other virtual threads.

Our code had quite a few synchronized locations — Guava Cache loading methods, Logback's async appender, and even some synchronized collection classes used internally by certain frameworks.

Fix

// synchronized causes virtual threads to be pinned
public synchronized String getConfig(String key) {
    return configMap.get(key);
}

// Switch to ReentrantLock
private final Lock lock = new ReentrantLock();

public String getConfig(String key) {
    lock.lock();
    try {
        return configMap.get(key);
    } finally {
        lock.unlock();
    }
}

But this wasn't the biggest headache — some synchronized blocks inside third-party libraries are simply impossible for you to change. For example, synchronized code blocks inside certain older database drivers or HTTP clients; these are the real ticking time bombs.

Data

Actual stress test data: Under frequent synchronized pinning, the throughput advantage of virtual threads dropped from 30% to almost zero, even worse than platform threads.

Scenario Throughput (vs Platform Threads) P99
No synchronized (pure compute+IO) +32% -42%
Small amount of synchronized +15% -18%
Frequent synchronized -5% +63%

Pitfall 2: ThreadLocal Memory Leaks

Phenomenon

After the service ran for a while, significant GC pressure appeared, and Old Gen kept growing. Investigation revealed that a large number of ThreadLocal objects were not being recycled.

Cause

In the JDK 21 virtual thread implementation, the virtual thread pool caches used virtual thread instances (the Loom team calls it a "carrier thread pool"). If code sets a ThreadLocal in a virtual thread but doesn't clean it up promptly, these objects will persist.

Our problems were mainly concentrated in:

  1. Spring Security's SecurityContextHolder — uses InheritableThreadLocal by default
  2. MDC (Mapped Diagnostic Context) — Logback's context propagation based on ThreadLocal
  3. ThreadLocals implicitly set in various interceptors/AOP

What's worse: The number of virtual threads can be far greater than platform threads, meaning the risk of ThreadLocal leaks is magnified by an order of magnitude.

Fix

# In Spring Boot, extra configuration is needed in virtual thread mode
spring:
  threads:
    virtual:
      enabled: true
  task:
    scheduling:
      pool:
        size: 10  # Be sure to set a reasonable thread pool size

Code level:

// Rely on automatic cleanup
SecurityContextHolder.setContext(ctx);
try {
    // Business logic
} finally {
    // Explicit cleanup
    SecurityContextHolder.clearContext();
}

But saying this is easy. Actually checking all ThreadLocal usage across the entire project and ensuring every single one has a try-finally cleanup is a massive undertaking. If you use third-party frameworks that implicitly set ThreadLocals, troubleshooting is even more painful.

Pitfall 3: Thread Pool + Virtual Threads = Negative Optimization

Phenomenon

We had a scenario: upstream sends a batch of requests (about 500-2000), and each request needs to make parallel calls to multiple downstream systems. Originally, we used CompletableFuture + a custom thread pool.

After switching to virtual threads, the code was simplified:

// "Simplified" writing after using virtual threads
List<Future<Result>> futures = tasks.stream()
    .map(task -> executorService.submit(task))
    .toList();

Looks fine, right? The problem lies with executorService.

Cause

If you still use a traditional thread pool (newFixedThreadPool, etc.) to run virtual threads, it's a complete disaster — the biggest advantage of virtual threads (lightweight, massive quantity) is completely neutered.

The correct approach should be to use Executors.newVirtualThreadPerTaskExecutor(): create a new virtual thread for each task, with no limit on quantity.

But then another problem arises: No limit doesn't mean truly unlimited. When your virtual threads are waiting on IO, they don't consume platform threads, but each virtual thread still occupies at least a few KB of memory. If you create 100,000 virtual threads simultaneously, the memory footprint alone approaches 1GB, and GC pressure increases significantly.

Lesson

// Wrong: Managing virtual threads with a platform thread pool
ExecutorService pool = Executors.newFixedThreadPool(200);
Future<?> f = pool.submit(virtualThreadTask);

// Wrong: Unlimited creation of virtual threads
ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();
for (int i = 0; i < 100_000; i++) {
    executor.submit(task); // Memory explodes directly
}

// Correct: Limit the concurrency of virtual threads
Semaphore semaphore = new Semaphore(1000);
ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();
for (Task task : tasks) {
    semaphore.acquire();
    executor.submit(() -> {
        try {
            task.run();
        } finally {
            semaphore.release();
        }
    });
}

Pitfall 4: Monitoring and Troubleshooting Tools Collectively Fail

Phenomenon

When problems occurred online, we habitually used Arthas to check thread status, and the result was:

[arthas]$ thread
Threads: Total: 12 (platform threads only)

Virtual threads simply don't show up as independent threads in Arthas! Troubleshooting relied entirely on guesswork.

The Real Pain

We later had to write a custom set of tools to aggregate virtual thread scheduling information into logs, barely restoring observability. This single task took two weeks of work.

Pitfall 5: Hidden Problems in Spring Boot

If your project uses Spring Boot like most people, you might also encounter these hidden problems:

Problem 1: Connector issues in Tomcat virtual thread mode

server:
  tomcat:
    threads:
      max: 200  # The meaning of this config changes in virtual thread mode

In platform thread mode, the Tomcat thread pool size determines the maximum number of concurrent requests. But in virtual thread mode, this value becomes the request queue length — because each request is handled by a virtual thread, not a platform thread. This semantic change caused a large number of requests to be queued during our stress tests, while the actual processing capacity was far from fully utilized.

Problem 2: Thread management in RestTemplate/WebClient

// In virtual thread mode, RestTemplate creates a new virtual thread for each request
// But improper connection pool configuration leads to many virtual threads waiting for connections
@Bean
public RestTemplate restTemplate() {
    HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
    factory.setConnectTimeout(5000);
    factory.setReadTimeout(5000);
    factory.setHttpClient(HttpClientBuilder.create()
        .setMaxConnTotal(200)        // This number needs to match the concurrency level
        .setMaxConnPerRoute(100)
        .build());
    return new RestTemplate(factory);
}

Virtual threads multiplied the concurrency several times over, but the connection pool didn't keep up, causing a large number of virtual threads to block on "waiting for HTTP connection." While they indeed don't consume platform threads while waiting, the long wait time still results in a poor user experience.

Problem 3: Database Connection Pool

The same logic applies — virtual threads allow your application to handle higher concurrency, but the upper limits of external resources like database connection pools, HTTP connection pools, and Redis connection pools remain unchanged. Ultimately, a large number of virtual threads block waiting for connection resources, and overall latency actually worsens.

So, Are Virtual Threads Any Good or Not?

After talking about so many pitfalls, I'm afraid of misunderstandings. Our conclusion is not "virtual threads are no good," but rather use them in the right scenarios, in the right way.

Scenarios Where We Finally Kept Virtual Threads

Scenario Kept? Reason
IO-intensive tasks (HTTP calls, DB queries) Kept Significant improvement + code simplification
CPU-intensive tasks Switched back to platform threads No benefit and pinning issues
High-throughput gateway Conditional use Requires thorough tuning + fixing synchronized
Data batch processing Kept Lots of IO waiting, clear benefits
Real-time trading system Switched back to platform threads Unacceptable P99 jitter

Current Conclusion

The real value of virtual threads is not "making programs run faster," but making the programming model simpler. With virtual threads, you can achieve asynchronous performance while maintaining a synchronous programming style. But this value comes with conditions:

  1. Your code/dependency libraries barely use synchronized
  2. ThreadLocal usage is standardized and cleaned up promptly
  3. You are willing to invest time in refactoring monitoring and troubleshooting tools
  4. The capacity of external resources (connection pools, etc.) can keep up with the concurrency of virtual threads

If you cannot meet the above four points, it's recommended to start with a small-scale canary deployment first, rather than rolling it out fully on core services directly.

Summary

It's 2026, and virtual threads are no longer a novelty. Various major companies have had both successes and failures in practice, but a consensus is forming: Virtual threads are a tool, not an architecture.

They cannot replace an understanding of concurrency models, cannot replace reasonable connection pool configuration, cannot replace good programming habits — but they can indeed make writing IO-intensive code easier.

Next time someone says "Let's just switch to virtual threads," I suggest you ask three questions:

How much synchronized is in your code? Is ThreadLocal cleanup standardized? Have the connection pools been scaled up?

If they can't answer, it's fine to upgrade the JDK to 21 to try it out first, but don't rush to enable virtual threads on the critical path.

Have you tried virtual threads in a production environment? What pitfalls have you encountered? Feel free to share your experience in the comments.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

jvmind_dev

The tools for analyzing virtual threads haven't kept up with development. As far as I know, jcmd <pid> Thread.dump_to_file -format=json threads.json can output virtual thread information, but you need to write your own tool to analyze it.