A ConcurrentHashMap Cached Embeddings to Save Pennies — and Nearly Killed an AI Chatbot at Launch

This isn't a textbook article. It's a real, bloody, tear-filled experience of stepping on a landmine, not sleeping well for 3 days, and almost getting called in by my boss.

1. That Afternoon, I Thought It Was Over

During the final round of stress testing before the project launch, I was sitting at my desk staring at the Grafana dashboard, feeling fairly calm—the test environment had run for a month without major issues, P99 was consistently around 2 seconds. Although it wasn't fast, the LLM itself took 2 seconds, so I thought it was acceptable.

Then 1000 concurrency hit—

P99 shot straight up to 5 seconds.

It wasn't a slow rise; it was the kind where you watch the curve "whoosh" straight up. I thought the monitoring was bugged, refreshed the page twice, and the number was still 5 seconds.

Even more terrifying, the curve wasn't smooth—it looked like an electrocardiogram, "pausing" at regular intervals, then recovering, then pausing again. User feedback said, "The chat froze, waited a long time for a reply, then suddenly it was fine again."

My first reaction: It's over, the LLM interface is broken.

I rushed to check the large model service's monitoring—a rock-solid 2-second response, not a single fluctuation.

So the problem was on our side. But my code didn't have any complex logic: just calling the LLM API, assembling messages, storing context... where did this 5-second lag come from?

At that moment, I had only one thought: I'm definitely taking the blame for this.

---

2. Starting the Hunt—Secrets in the GC Logs

I steeled myself and dug through the GC logs. I thought it would just be ordinary Minor GC, but then I saw this line:

[Full GC (Allocation Failure) 3.2G->2.8G(4G), 2.5 secs]

A 2.5-second Full GC, 3.2G before reclamation, 2.8G after—only 400MB reclaimed?

Happening 3-5 times an hour, each pause lasting 2-3 seconds. This was the damn source of that "electrocardiogram"! The application paused for 2-3 seconds, the user felt the lag, GC finished and it recovered, then after a while another GC, another lag...

And the old generation occupancy was still 85% after reclamation—meaning a large number of objects simply couldn't be reclaimed. This wasn't normal GC behavior; this was a precursor to a memory leak.

My heart sank: It's over. It's not an interface problem; it's a problem I wrote into my own code.

---

3. jmap -histo: The Culprit Surfaces

Just looking at GC logs only lets you guess the direction; you need to see specifically what is occupying memory.

jmap -histo:live <pid> | head -20

When the output came out, I nearly coughed up blood:

 num    #instances   #bytes  class name
   1:       50000    209715200  [F               ← float array! 200MB!
   2:       10000    104857600  ConversationContext  ← Session objects 100MB
   3:       10000     52428800  ConcurrentHashMap$Node ← HashMap nodes 50MB

Float arrays taking up 200MB? Where did my project have so many floats?

Then it hit me—my Embedding cache.

Each session stored Embedding vectors for multiple rounds of conversation, 1024-dimensional floats, one is 4KB. To save the cost of calling the Embedding API (a few cents per call), I locally cached all computed vectors.

I did the math:

Data Item	Per Session Usage	1000 Users
Conversation History (20 messages)	~20 KB	~20 MB
Tool Cache (5 results)	~5 KB	~5 MB
Embedding Cache	~100 KB	~100 MB

100KB multiplied by 1000 users is 100MB. Add conversation history and other data, one session is 130KB, 1000 concurrent sessions is 130MB heading straight for the old generation.

Saved a few cents on API costs, at the expense of Full GC wrecking the entire system's response time. No matter how you calculate it, this deal was a loss.

---

4. MAT Deep Dive: The Full Reference Chain Exposed

jmap -histo only gives a rough picture. To confirm exactly who was referencing these objects and why they couldn't be reclaimed, I had to export a HeapDump and analyze it with MAT.

jmap -dump:format=b,file=/tmp/heapdump.hprof <pid>

Opening the hprof file in MAT, I immediately saw the GC Roots reference chain:

SessionManager.sessions (ConcurrentHashMap)
  → ConversationContext
    → embeddingCache (HashMap)
      → float[1024] × N

The entire chain was crystal clear:

1. SessionManager holds sessions, this ConcurrentHashMap
2. Each sessionId corresponds to a ConversationContext
3. Each ConversationContext contains an embeddingCache (HashMap)
4. embeddingCache stores multiple 1024-dimensional float arrays

Where's the problem? Objects in the ConcurrentHashMap are reachable from GC Roots. As long as the session is alive, these Embedding caches can never be reclaimed.

Sessions last 10-30 minutes, these objects just linger in the old generation, accumulating more and more until the old generation is full → Full GC → low reclamation rate → fills up again → another Full GC...

A vicious cycle.

At that moment I understood: I had forcefully stuffed things into the JVM heap that didn't belong there.

---

5. Emergency Stopgap: Keep the System Alive First

The boss was already pressing: "When can it go live?" I couldn't say "I'm troubleshooting a JVM issue." I had to stop the bleeding first.

First Move: Limit Cache Quantity

// ConversationContext.java
private static final int MAX_EMBEDDING_CACHE = 500;

public void putCachedEmbedding(String textHash, List<Float> embedding) {
    if (embeddingCache.size() >= MAX_EMBEDDING_CACHE) {
        // Exceeded limit, clear half
        int removeCount = MAX_EMBEDDING_CACHE / 2;
        Iterator<String> it = embeddingCache.keySet().iterator();
        while (removeCount-- > 0 && it.hasNext()) {
            it.next();
            it.remove();
        }
    }
    embeddingCache.put(textHash, embedding);
}

This is the crudest simplified LRU—when full, chop half. Not elegant, but effective.

Second Move: Increase Heap

# From 4G to 8G
-Xmx8g -Xms8g

A larger heap means more old generation space, naturally lowering the frequency of Full GC triggers.

With these two moves, Full GC dropped from 5 times per hour to 1-2 times per day, and P99 returned from 5 seconds to around 3 seconds.

The system was barely usable, but I knew this wasn't a fundamental solution—a larger heap only delays the problem, and cache limits only treat the symptoms. The root problem remained: Session data shouldn't be in the JVM heap.

---

6. The Fundamental Solution: Migrate to Redis

After stopping the bleeding, I spent a weekend doing the real refactoring—replacing ConcurrentHashMap with Redis.

Honestly, I hesitated at first: Redis adds a network call layer, serialization/deserialization has overhead, wouldn't it be slower?

But thinking it over, my bottleneck wasn't "slow session reads" at all; it was "GC pauses causing all requests to freeze." Even if Redis took an extra 1 millisecond per read, it's infinitely better than a 2.5-second GC pause.

@Component
public class SessionManager {

    @Autowired
    private StringRedisTemplate redisTemplate;

    private final Gson gson = new Gson();
    private static final String SESSION_KEY = "ai:session:";

    public ConversationContext getOrCreate(String sessionId, ...) {
        String key = SESSION_KEY + sessionId;

        // 1. Check Redis
        String json = redisTemplate.opsForValue().get(key);
        if (json != null) {
            return gson.fromJson(json, ConversationContext.class);
        }

        // 2. Create new, write to Redis (auto-expire in 30 minutes)
        ConversationContext ctx = new ConversationContext(sessionId);
        ctx.setUserId(userId);
        redisTemplate.opsForValue().set(key, gson.toJson(ctx),
                30, TimeUnit.MINUTES);
        return ctx;
    }
}

The best part of the Redis solution is TTL auto-expiry.

Previously with ConcurrentHashMap, I had to write my own scheduled tasks to clean up expired sessions—code full of removeIf, scheduleAtFixedRate, and always worrying about not cleaning thoroughly. With Redis, just expire 30min, and it disappears automatically when the time comes. Even session renewal is a single expire line.

Dimension	ConcurrentHashMap	Redis
Capacity	Limited by JVM heap (4~8G)	Independent process, scale arbitrarily
Distributed	Not supported	Natively supported
Restart	All data lost	Optional persistence
Expiry Cleanup	Hand-written scheduled tasks	TTL auto-expiry
Session Renewal	Hand-written logic	One line expire

---

7. Final Results—Finally Able to Sleep Well

After migrating to Redis, running stress tests again:

Full GC: 0 times/day ✅
P99 Latency: Dropped from 5 seconds to 2.2 seconds ✅ (LLM itself takes ~2 seconds, basically just model latency now)
Old Generation Occupancy: Stable below 30% ✅
Supports Horizontal Scaling: Multiple instances share Redis ✅

Looking at that flat P99 curve on Grafana, I was finally at ease.

3 days of poor sleep, worth it.

---

8. Review: JVM Tuning Isn't About Memorizing Parameters

This experience made me truly understand one thing: JVM tuning isn't about memorizing parameters; it's a data-driven closed-loop process.

My Four-Step Method

1. Baseline Test: Start with default config → stress test → record GC frequency, P99, heap usage rate
2. Analyze Bottleneck: GC logs (GCEasy online analysis) → memory distribution (jmap + MAT) → thread status (jstack / Arthas)
3. Targeted Tuning:
   • Frequent Minor GC → increase young generation
   • Frequent Full GC → increase heap + check for memory leaks
   • GC pause too long → switch collector (G1 → ZGC)
   • Memory leak → MAT analyze HeapDump, fix code
4. Verify Effect: Stress test again → compare to baseline → observe for 1-2 days to confirm stability

If no improvement, rollback, change direction, and retry. Don't stubbornly stick to one direction.

A Few Pits I Fell Into, Sharing With You

Pit 1: Xms does not equal Xmx

Many people set -Xms512m -Xmx4g in dev environments, thinking "save resources first, expand when needed." But never do this in production—JVM dynamically expanding the heap at runtime involves applying for memory + copying data, triggering STW pauses. Production needs predictability, so -Xms and -Xmx must be set to the same value.

Pit 2: Bigger Young Generation Isn't Always Better

Initially, I thought a larger young generation = more objects reclaimed in the young generation = better. But in my project, ConversationContext objects are long-lived (user chats last 10-30 minutes). They survive every Minor GC, their age increases rapidly, and they are quickly promoted to the old generation.

Large young generation suits short-lived objects (created and die immediately). Long-lived objects ultimately end up in the old generation. So leaving more space for the old generation is more important—young generation at 1/3 (smaller side), old generation at 2/3 (larger side) is more reasonable in scenarios with many long-lived objects.

Pit 3: GC Logs Are Not Decorations

Many people go live without even enabling GC logs, only to find they have no data to look at when problems arise. Production environments must enable them:

-Xlog:gc*:file=/data/logs/gc.log:time,uptime,level,tags:filecount=5,filesize=10m

Also configure HeapDump:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/logs/heapdump.hprof

Automatically exporting a heap dump on OOM is much better than manual jmap.

Pit 4: Arthas is a God-Tier Tool for Online Troubleshooting

The scariest thing about online issues is "not being able to see." Arthas allows you to view JVM info, memory distribution, method latency, and flame graphs online, without restarting or extra deployment:

# View instance count of a class
sc -d com.xxx.ConversationContext
# → instances: 1234 ← hits the nail on the head

# View real-time method latency
trace com.xxx.AiCustomerServiceEngine process
# → shows the time cost of each step

# Flame graph to find CPU hotspots
profiler start
# wait 30 seconds
profiler stop --format html --file /tmp/flamegraph.html

---

9. Decision Quick-Reference Table

What You Want To Do	What To Look At	How To Tune
Choose heap size	Total machine memory	Total memory × 70%
Choose young generation	Object survival time	Short-lived → large young gen; Long-lived → small young gen
Choose collector	Latency requirement + heap size	G1 (general) / ZGC (ultra-low latency)
Find GC problems	GC logs	GCEasy online analysis
Find memory problems	HeapDump	MAT analysis
Find CPU problems	jstack	Find hotspot threads/deadlocks
Online troubleshooting	Arthas	dashboard / trace / profiler

---

10. A Few Final Words From the Heart

The biggest lesson from this experience isn't how to configure JVM parameters—those are everywhere online. The real lesson is: Don't put things in the JVM heap that don't belong there.

Session data, cache data, state data—these things naturally belong in external storage (Redis, databases). The JVM heap is for running business logic, not for acting as a database.

When I initially used ConcurrentHashMap to store sessions, writing the code was indeed satisfying—thread-safe, zero network overhead, zero serialization cost. But the price was: heap memory pressure, GC burden, no distributed support, data loss on restart. These costs are invisible at low concurrency but become fatal flaws at high concurrency.

Shortcuts taken during design are debts to be repaid after launch.

---

If this article helps you avoid one pitfall, a like would make me very happy. Questions welcome in the comments; let's walk this minefield together.