跪拜 Guibai
← Back to the summary

Huolala's JDK 21 Migration: Framework First, AI-Assisted Rewrites, and 12 Silent Breakages

JDK 21 at Scale: Practice on the Ground

Author: Huolala Technology Tags: Backend

JDK 8 carried Huolala through an entire generation of microservices; more than a thousand online services still run on it. Moving to JDK 21 means the runtime, Spring, and a batch of middleware all have to change with it. Changing java.version is one line. The hard part is that after compilation succeeds, the behavior of many dependencies has quietly changed. This time, the framework absorbed the compatibility cost first, AI handled the repetitive code changes in bulk, and business projects switched parent and finished up against a checklist. Base moves first; business moves later.

1. Background

image 2.png

Upgrade benefits

Benefit Description
Lifecycle Back to mainstream LTS, with long-term security patches and root certificate updates. Spring Boot 2.x stopped open-source maintenance long ago; staying on the old stack means patching vulnerabilities yourself.
Ecosystem Open-source frameworks, cloud vendor JDKs, community solutions, and AI programming tools already treat JDK 21 and the new Spring generation as the default baseline. New capabilities, examples, and subsequent LTS versions are released along this line; external experience can land directly in the project without maintaining a separate branch for the old stack.
Resource efficiency The JVM correctly recognizes the container memory and CPU quota instead of estimating from the host. At the same business volume, production measurements show CPU down 10% and memory down 50%.

Upgrade challenges

Moving to JDK 21 almost inevitably brings a full-stack replacement: Spring Boot 3, Jakarta EE, new-generation connectors and serialization libraries, plus a batch of internal second-party packages. If only the JDK is upgraded and the transitive dependencies and runtime semantics do not move along with it, compilation often still passes; classes, annotations, and serialization only mismatch at startup or at the first request.

More hidden problems appear during release and gray release. Auto-configuration moved from spring.factories to AutoConfiguration.imports; a custom starter that only writes the former silently fails. Connectors, RPC, and monitoring components also have different default behavior from 2.x; latency, metrics, or traces drift slowly as traffic ramps. The thread model and container CPU/memory visibility behave the same way; copying old parameters only exposes problems after traffic has switched.

Upgrade approach

Upgrade method Each project upgrades itself Framework moves first
Version management Each manages versions in its POM BOM centralizes; business does not write versions
Jakarta / middleware / RPC Each project changes them separately Done in the framework first
Code changes Sample code hand-modified, compatibility issues remembered manually AI rewrites in bulk, scans against known compatibility issue list
What business does Find dependencies one by one Switch parent, align baseline by checklist

All Java services at the company are built on JAF, so most compatibility work can be done by the framework before business projects touch it. Overall approach:

After switching parent, business projects still need to align javax, custom starters, internal components, and Lombok to the same baseline according to the checklist at the end of Chapter 2; otherwise issues the framework already solved reappear in business projects.

2. Framework Replacement: Base First, Business Later

If more than a thousand services each upgrade themselves, the effort is hard to close out. So this replacement is mainly carried by the framework layer. Framework versions are divided by major version: 2.x corresponds to JDK 8 and Spring Boot 2; 3.x corresponds to JDK 21, Spring Boot 3.5, and Jakarta. JAF, Hermes, and Monitor, the three core repositories, upgrade together and align versions; business projects only declare dependencies without writing version numbers.

image 3.png

Component Role What this replacement does
lalaframework (JAF) Distributed application framework: config, cache, ORM, message, SOA, governance, and other Starter systems Compile baseline to JDK 21, javax fully moved to jakarta, dependencies centralized through a BOM
hll-hermes In-house SOA/RPC framework: JSON-RPC / gRPC, Consul discovery, routing and governance RPC aligned with Spring 6, version-bound with JAF
lala-jaf-monitor Monitoring: Prometheus metrics and SkyWalking traces, agent and non-agent modes Metrics and traces moved to new engines, observability uninterrupted during gray release

Hermes and JAF are strongly version-bound (hermes.version = lala.release.version), so cross-generation mixing does not happen. The 2.x and 3.x lines are maintained in parallel: new projects use the 3.x scaffold directly, existing services migrate at their own pace without a one-time cutover; capabilities verified on 2.x, such as Kafka and exemplar support, are also backported to 3.x.

Dependencies Raised as a Group

The first step is to raise the whole dependency group, locked by BOM versions, with a lot of dependency compatibility adaptation in between.

Two categories need focused attention:

Whether Spring-managed component versions fit internal components

Example: Redis. Does the internal Datamesh/redis version fit, including protocol and commands? Others such as MySQL and Elasticsearch all need to fit the company's internal server versions.

Whether a component itself has bugs

Example: httpcore5, managed by Spring 6.2.7. Version 5.4 has a bug where the default tcpNoDelay:false does not apply, adding about +40ms latency to large RPC payloads; only upgrading to 5.4.2 fixes it.

The main dependency version comparison is below. Business projects should not override these versions in their POMs:

Dependency 2.x 3.x
JDK 8 21
Spring Boot 2.2.6 3.5.0
Spring Framework 5.2.5 6.2.7
Servlet API javax 4.0 jakarta 6.0
Micrometer 1.5.x 1.15.0
Fastjson 1.2.x 2.0.x
MyBatis / MyBatis-Plus 3.5.4 / 3.3.0 3.5.19 / 3.5.12
MySQL Connector 5.1.48 9.2.0
gRPC 1.41.1 1.73.0
Jackson 2.10.3 2.19.0
Httpclient5 4.x 5.x

Internal companion components must be the same generation as the framework; mixing a 2.x framework with 3.x components is not supported. During upgrade review, first check the dependency tree for conflicts, then look at business code diffs.

JAF: Middleware Wrappers on a New Base

Hermes: Spring 6 Adaptation for the RPC Base

Pitfalls found after upgrade: after the HttpClient5 upgrade, large RPC latency rose about +40ms, caused by Nagle and Delayed ACK interacting. Explicitly enabling tcpNoDelay and upgrading httpcore5 to 5.4.2 fixed it completely; the connection pool default policy was also aligned back to the pre-upgrade behavior.

Monitor: Metrics and Traces on a New Engine

Business Upgrade Checklist

After the framework finishes the common parts, what each business project must change converges to the following five items. For what to test after changes, see the regression checklist at the end of Chapter 3.

3. Pitfall Log: Compilation Is Green, Semantics Have Changed

In large-scale upgrades, the most time-consuming problems are where compilation is fully green but semantics have changed. The 12 pitfalls below were all really hit in this upgrade. They are split into dependency/contract issues and release/resource-model issues. Each one is written as symptom, cause, and action. At the end of the chapter, there is a regression checklist for direct use.

image 5.png

Dependency and Contract: Interface Unchanged, Behavior Changed

Incidents 01 to 03 are dependency and build issues: old versions and old wrappers fail directly on the new stack. Incidents 04 to 08 are contract and semantic issues that the compiler cannot catch at all; they can only be covered by regression testing.

image.png

image 4.png

Release and Resource Model: Pitfalls That Only Explode in the Gray-Release Window

Incidents 09 to 12 are related to release mode and machine specifications. Some appear only in the gray-release window; others can only be reproduced on production-spec machines and are hard to find in non-production environments.

image 1.png

Regression Checklist

Go through each item during regression. The corresponding pitfall number is in parentheses.

4. Feature Application: Virtual Threads and Language Features

Virtual Threads: Choose the Scenario First

Platform threads map one-to-one to OS threads, and both creation and switching are expensive. Virtual threads are scheduled by the JVM; when blocked on HTTP, JDBC, Redis, or RPC, they unmount from the carrier thread, which can switch to other virtual threads. So they suit synchronous code with lots of waiting and little computation. IO concurrency can be limited with Semaphore or a connection pool; there is no need to configure a tiny platform thread pool just for rate limiting.

image 9.png

These scenarios continue to use platform threads: CPU-intensive encryption and decryption, and large JSON serialization; virtual threads do not make computation faster. Scheduled tasks, XXL-JOB, and Kafka consumer threads are managed by their respective frameworks, and their concurrency is already limited by task count or partition count, so switching them to virtual threads brings little benefit.

image 7.png

With Tomcat, enabling spring.threads.virtual.enabled switches the whole request onto a virtual thread. Before enabling it, confirm that Filters and interceptors do not do blocking IO inside synchronized. The request itself does not break Trace; only when you create new virtual threads inside a request do you need the Helper.

Two JAF Integration Modes

VirtualThreadContextHelper

When the parent thread hands a task to a new virtual thread, ThreadLocal does not propagate automatically. Inside the same virtual thread, blocking IO and carrier switching do not require another wrapper. The Helper does two things before submission:

  1. RunnableWrapper.of captures the SkyWalking Trace.

  2. ContextAwareRunnable.of carries the business ContextHeaders.

runVirtual corresponds to Thread.startVirtualThread.

newVirtualThreadExecutor corresponds to an unbounded executor that creates one new VT per task, like ThreadPerTaskExecutor.

// Native (usable for standalone scripts without Trace requirement)
try (ExecutorService ex = Executors.newVirtualThreadPerTaskExecutor()) {
    ex.submit(() -> doWork());
}

// Framework recommended (business traces always use this)
VirtualThreadContextHelper.runVirtual(() -> doWork());
CompletableFuture<String> f = VirtualThreadContextHelper.supplyVirtual(() -> loadData());
ExecutorService ex = VirtualThreadContextHelper.newVirtualThreadExecutor("biz-virtual-");

Dynamic Thread Pool

After enabling virtualThread=true, the dynamic thread pool only changes the way threads are created. The executor is still a ThreadPoolExecutor; core size, max size, queue, and rejection policy all remain effective, and the created Worker threads are all virtual threads. The scheduling pool DtpScheduledExecutor ignores this switch.

@Bean
public ThreadPoolExecutor virtualExecutor() {
    return ThreadPoolBuilder.newBuilder()
            .threadPoolName("virtualExecutor")
            .corePoolSize(20)
            .virtualThread(true)
            .maximumPoolSize(100)
            .queueCapacity(2048)
            .buildDynamic();
}

It differs from newVirtualThreadPerTaskExecutor(): that one creates a new VT per submitted task and has no queue and no core/max. DTP core/max limits the Worker virtual thread count, not the carriers. Carriers are provided by the JVM on demand based on CPU core count, with a default upper limit around max(cores, 256), shared with Tomcat and the Helper. In IO scenarios, raising DTP max increases concurrency.

Language Features

Features in JDK 21 that can be used immediately are the finalized pattern matching, Record deconstruction, sequenced collections, and text blocks. They do not affect performance; the benefit is fewer casts and less boilerplate. With sealed interfaces, they can also catch missing branches at compile time.

Can go into production: pattern matching, Record deconstruction, sequenced collections, text blocks.

Disabled in production: String Templates, Structured Concurrency, and other features that are not finalized. Do not add --enable-preview.

Pattern Matching: One Less Cast

The instanceof type check can bind a variable directly; this has been available since JDK 16. Switch type patterns were finalized in JDK 21. Both remove casts and repeated null checks.

if (factory instanceof NamedThreadFactory named) {
    return named.getNamePrefix();
}

if (dataSource instanceof ItemDataSource itemDataSource) {
    return itemDataSource.getRealDataSource();
}

switch suits one entry point dispatching to multiple types. When a sealed interface is switched on without a default branch, a missing branch is a direct compile error. Two easy pitfalls:

static String describe(Object value) {
    return switch (value) {
        case null -> "empty";
        case String s when s.isBlank() -> "blank";
        case String s -> "text:" + s.length();
        case Integer i -> "int:" + i;
        default -> value.getClass().getSimpleName();
    };
}

Record: Data Carrier Only, Not a Bean

Suitable for: read-only data objects, cache keys, route keys assembled from a few fields.

Keep using class for: MyBatis entities, config with setters, Spring Beans.

When used with switch, record fields can be deconstructed directly:

public record RouteKey(String tenantId, String table) {}

static String shard(Object key) {
    return switch (key) {
        case RouteKey(String tenant, String table) -> tenant + ":" + table;
        case String s -> s;
        default -> "default";
    };
}

Sequenced Collections: Official getFirst / getLast

JDK 21 adds unified getFirst() and getLast() for ordered collections such as List and LinkedHashSet, so get(0) and get(size - 1) are no longer needed. LinkedHashMap has firstEntry() and lastEntry(). Calling getFirst() on an empty List throws NoSuchElementException, so check emptiness first. reversed() returns a reverse view, not a copy; use new ArrayList<>(list.reversed()) when you need a snapshot.

List<String> steps = new ArrayList<>(List.of("auth", "route", "invoke"));
steps.addFirst("warmup");
String last = steps.getLast();
List<String> reversed = steps.reversed(); // view, not copy

LinkedHashMap<String, Integer> order = new LinkedHashMap<>();
order.putLast("tail", 1);
Map.Entry<String, Integer> first = order.firstEntry();

Text Blocks: Test Samples and Fixtures, Not Runtime SQL

Text blocks (""") suit JSON fixtures in unit tests and error message templates. Runtime SQL assembled with user input still has injection risk; that has not changed.

String payload = """
        {"orderId": "%s", "status": "CREATED"}
        """.formatted(orderId);

5. Runtime Governance: Let the JVM See the Real Container

The runtime benefits of the upgrade reach production through three things: choosing GC by latency target, making the JVM read the container's real memory and CPU, and troubleshooting with aggregate metrics first instead of only dumps. Conclusion first: use G1 by default; evaluate generational ZGC when interfaces are latency-sensitive and the heap is large. Set heap size by percentage; do not hardcode -Xmx.

How to Choose GC

JDK 21 defaults to G1, and most business should start there. Generational ZGC has shorter pauses but uses more memory and CPU. Only one collector can remain in the startup parameters; when -XX:+UseG1GC and -XX:+UseZGC appear together, the JVM fails to start. When switching, delete the old parameter.

Dimension G1 (default) Generational ZGC
Suitable heap 2G to 16G most common; larger can also run More worthwhile above 8G; small heaps get little benefit
Pause target Usually 50 to 200ms; target is not a hard constraint Usually within a few milliseconds; more stable
Throughput Better, lower CPU overhead Slightly worse; pays for latency stability
Additional memory overhead Relatively small Larger; container needs extra headroom
When to use JAF / middleware default; start here Evaluate when slow requests cannot be solved and heap keeps growing 
java -XX:+UseG1GC \
     -XX:MaxGCPauseMillis=200 \
     -XX:InitiatingHeapOccupancyPercent=45 \
     -XX:+ParallelRefProcEnabled \
     -XX:+ExitOnOutOfMemoryError \
     -Xlog:gc*,gc+heap=info:file=/var/log/gc.log:time,uptime,level,tags:filecount=5,filesize=50M \
     -jar app.jar

Generational ZGC does not need many tuning items; giving it enough heap and CPU matters more. With only 1 core in the container, collection drags for a long time, which looks like no full-heap collection but interface timeouts. After go-live, compare three things: whether slow requests dropped, whether CPU rose obviously, and whether container memory is closer to the limit. The switch parameters are -XX:+UseZGC and -XX:+ZGenerational. ZGC itself uses more memory than G1; start with a heap ratio from 65%, and adjust to around 70% when GC pressure is high.

Container Memory and CPU

JDK 21 reads the container memory limit and CPU quota instead of estimating from the host. Previously, with a 64-core 128G host and a 2-core 4G container, the JVM calculated default heap and GC threads from the host, easily exceeding the container quota. After switching to a JDK 21 image, most such problems disappear automatically; what remains to adjust manually is the heap percentage and off-heap reservation.

image 6.png

Memory slice Who uses it How to reserve it in a container
Java heap Objects, G1/ZGC regions MaxRAMPercentage: G1 70 to 75, ZGC 65 to 70
Metaspace / CodeCache Classes, JIT Spring Boot 3 services often use 256M to 512M; do not ignore
DirectMemory / Netty NIO, RPC, gateway Explicitly set -XX:MaxDirectMemorySize; the default can approach heap size
Thread stacks Carrier platform threads plus virtual threads Virtual thread stacks are small, but carrier threads still count as platform threads
GC itself Collector ZGC is clearly higher than G1, so ZGC heap ratio should not be opened above 85%

Use the company's unified JDK 21 base image. Do not compile with 21 and run with 17. Below is a startup command for a 4G container with G1 that can be copied directly into the deployment template. During the initial virtual thread rollout, keep pinned thread tracking on, then turn it off after the hot path is confirmed clean:

java -XX:+UseG1GC \
     -XX:MaxRAMPercentage=70.0 \
     -XX:InitialRAMPercentage=35.0 \
     -XX:MaxDirectMemorySize=512m \
     -XX:+ExplicitGCInvokesConcurrent \
     -XX:+ExitOnOutOfMemoryError \
     -XX:+HeapDumpOnOutOfMemoryError \
     -XX:HeapDumpPath=/var/log/heapdump.hprof \
     -Djdk.tracePinnedThreads=short \
     -Xlog:gc*,gc+heap=info:file=/var/log/gc.log:time,uptime,level,tags:filecount=5,filesize=50M \
     -jar app.jar

Observability: Aggregate First, Then Drill Down

Previously, one thread dump could show who was stuck. With virtual threads, the count can reach hundreds of thousands; a full dump consumes disk and cannot be read manually, and most threads are only waiting on IO. The troubleshooting order has to change:

Tool What to look at How to use it
GC logs Pauses, old-generation occupancy, allocation speed Compare 30 minutes after release; with ZGC focus on allocation being stuck, not pause time
JFR GC, locks, pinned virtual threads, IO jcmd JFR.start name=onboarding settings=profile dumponexit=true
Pinning tracking Virtual thread doing IO inside synchronized -Djdk.tracePinnedThreads=short
Thread snapshot Carrier threads and a small number of business stacks jcmd Thread.dump_to_file -format=json /tmp/threads.json; do not paste hundreds of thousands of lines into a group
Dynamic thread pool metrics Pool size, rejections, virtual thread count Metric prefix hllci_dtp.*
Monitor Trace across virtual threads Carrier switching on the same virtual thread does not break the chain on 8.x; cross-task work must use the Helper

After go-live, first run a comparison. It is not enough that the process is alive:

6. Future Outlook

After the upgrade, these areas continue to evolve:

Direction Now Next
Concurrency model Virtual threads enabled by scenario; cross-task Trace and context through the Helper IO-heavy chains default to virtual threads; the framework catches Trace and ThreadLocal at task submission
Locks and pinning Hot paths replace synchronized with ReentrantLock; early rollout uses pinning tracking as a gate From JDK 24, synchronized no longer pins carrier threads (JEP 491); after the next LTS, JDK 25, new code no longer needs to avoid synchronized deliberately
GC G1 by default; large-heap services evaluate generational ZGC separately Generational ZGC covers more large-heap, low-latency services; G1 remains the default for most business
Runtime baseline Heap percentage, JFR, and thread pool metrics are configured by each service using this document as reference Container-aware heap percentage is written into the image baseline; JFR and thread pool metrics are written into release gates

Every pitfall hit and every parameter tuned in this upgrade eventually sinks into the framework and the base image. At the next runtime upgrade, the best business should still only need to change one line of parent.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

AI向善

For JDK, I first added a local cache layer, which cut the number of requests by about half; compared with the approach in the article, each has its own trade-offs.