Five-Stage Agent Resuscitation Manual: Keeping Your AI Alive When Concurrency Spikes

When you run an Agent locally, you think it's Zhuge Liang: it knows everything, calls tools with ease, and even writes your weekly report.

Once you push it to production, as soon as concurrency rises a bit, it turns into an "artificial idiot": the page keeps spinning forever, the backend logs turn red, 504 Gateway Timeout, ReadTimeout, and 429 Too Many Requests are having a party, and finally the service just dies.

An Agent is an extremely fragile "chain of traps" system: it not only counts tokens and queries vector stores, but also fires HTTP requests to call tools, then makes the LLM chat. If any single link in the chain fails, the whole thing collapses like dominoes.

To pull your Agent out of the ICU, first understand how it kills itself across these 5 stages.

Full-Link Accident Map

Every time a user sends a message, the Agent's internal journey looks like this:

[Frontend user eagerly asks]
       │
       ▼
【Stage 1: Prompt Assembly】(dig through chat history, check token count)
       │
       ▼
【Stage 2: RAG Retrieval】(convert text to vectors, search document store, rerank)
       │
       ▼
【Stage 3: LLM First Reasoning】(LLM is stuck: should I answer directly or call a tool?)
       │
       ▼
【Stage 4: External Tool Execution】(query business DB, fetch web pages, call third-party APIs)
       │
       ▼
【Stage 5: LLM Final Output】(compress tool output into an answer, deliver to user)

Stage 1: Prompt Assembly - Tired Out Before Leaving Home

Many think splicing system prompt and user input with f"{sys_prompt} + {user_input}" takes only 0.1ms. But in production, many turn this step into a CPU meltdown scene.

1. How It Fails

• Relentless DB calls: Every time you assemble context, you faithfully go to MySQL to fetch the last 50 chat messages. Under concurrency, the DB connection pool is depleted instantly, new requests wait in a desperate queue, and simply "borrowing a car" takes 5 seconds.

• Tokenizer (tiktoken) burns CPU: To avoid exceeding the model window, many write a while loop: run tiktoken to count tokens, cut one history item, count again, cut again... Tokenization is extremely CPU-intensive! With a few dozen more concurrent requests, the server CPU hits 100% and the event loop of the whole process freezes.

2. Rescue Guide

• Don't measure fabric with a millimeter ruler: When truncating history, stop using the tokenizer to count obsessively! Instead, estimate length by multiplying Chinese character count by 1.3~1.5, trim roughly, and do a final validation before sending to the model.

• Use cache, don't fight disk and DB: Put session history in Redis, and keep static prompt templates in memory at service startup. Don't read a local file every request.

Stage 2: Knowledge Recall (RAG) - The Vector Store Collapses While Searching

RAG is the Agent's "external brain", but under high concurrency that brain easily suffers a stroke.

1. How It Fails

• Embedding API surrenders first: Under concurrency, all texts line up to call cloud APIs to convert to vectors; OpenAI throws 429 Too Many Requests in your face. If you deploy an open-source model on your own GPU, VRAM blows out and queues wait forever.

• Vector store in deadly computation: Vector search is essentially geometric distance calculation in high-dimensional space. Under concurrency, the vector store server CPU maxes out and latency spikes from 20ms to 20 seconds.

• Rerank chokes on too much: You painstakingly retrieve top 100 chunks, then feed them to a high-latency Cross-Encoder model for reranking. The model says, "You're embarrassing me, Fatty." It just freezes.

2. Rescue Guide

• Laziness is a programmer's virtue (use cache): For frequent questions, don't keep hitting the vector store. Use Redis for exact hash cache, plus semantic cache (like GPTCache). If similarity reaches 95% or more, directly reuse the previous result.

• Batch and filter, don't push the whole load: Use BM25 keyword matching to narrow candidate chunks to under 20 before reranking. Also lower the vector store's retrieval precision parameters (like HNSW efSearch). Sacrifice 1% precision to multiply QPS several times - totally worth it.

Stage 3: LLM First Reasoning - Noisy Neighbors in the Shared Pool

Here, the LLM decides what to do. The pitfalls here are often not your bad code, but that you don't know how the vendor is screwing you.

1. How It Fails

• Public cloud shared pool "snatching veggies": You're buying a few cents per thousand tokens of public API, which means you're squeezing into a public server with thousands of others. If a neighboring customer suddenly floods traffic, the vendor's cluster queue stretches long, and your request's time-to-first-token (TTFT) jumps from 800ms to 30 seconds.

• You stuff a dictionary into the LLM: Registering dozens of Tool Schemas at once means the model has to read all those instructions each round, draining compute and slowing generation like an old ox pulling a broken cart.

2. Rescue Guide

• The clever rabbit has three burrows; prepare backups: Use LiteLLM or OneAPI gateway, hold multiple backup keys for polling. If the primary model doesn't emit a token within 5 seconds or returns 429, immediately degrade seamlessly to a backup vendor or self-hosted model.

• Tool pruning: Don't throw all tools at it at once. Use a lightweight rule to detect the current intent and only mount 3-5 relevant tools.

Stage 4: External Tool Execution - Crew Drops, Captain Dies Too

After the LLM thinks, it commands the Agent to call real-world tools. This is often the worst accident area.

1. How It Fails

• No timeout on tools: Some external API call lacks Socket Timeout (default is never or tens of seconds). If the remote server hangs, your Agent coroutine sits there waiting, consuming all concurrent connections.

• ReAct infinite loop (ghost walking): The tool reports an error, but the model doesn't believe it: "Must be the wrong pose, I'll try again." Tool errors again, model retries... After a few rounds, it burns through all max steps and timeout limits.

• Sitting in a row (serial calls): You have weather, calendar, and inventory lookups with no dependencies, but code uses serial await, turning a 3-second task into 9 seconds.

2. Rescue Guide

• Set hard timeouts: Give each tool its own time limit (internal APIs 2 seconds, scrapers 5 seconds). If it exceeds, cut it off immediately and feed "timeout" back as error text to the model: "Timed out, stop waiting, change approach or tell the user."

• Async parallel, take off: For independent tools, use Promise.all (Node.js) or asyncio.gather (Python) to fetch concurrently.

• Kill infinite loops: Limit max execution steps to 3-5. If you detect the same tool called twice with identical parameters and errors, force a circuit break and let the LLM summarize with available information.

Stage 5: LLM Final Output - Gateway Waits 60s, Cries and Returns 504

Tools return data, the LLM prepares its final answer. Don't trip at the finish line.

1. How It Fails

• Holding a big burst kills the gateway: Without streaming, the Agent silently generates a long response. The outer Nginx or load balancer waits 60s and sees no response, assumes the backend is dead, and returns 504 Gateway Timeout.

• Tool spits out garbage: External API returns several thousand characters of full HTML or large JSON, and code feeds it to the model for re-processing without cleaning. Context explodes instantly; the model gasps just to swallow it, and generation speed tanks.

• User left, you're still punching thin air: The user gets impatient and closes the page, but the backend never listens for cancellation signals, so the server keeps calling the LLM and burning money, occupying concurrency slots.

2. Rescue Guide

• Must use streaming (SSE / WebSocket): Don't hold it in! As the LLM emits a token, push it to the frontend immediately. As long as data keeps moving on the network channel, the gateway won't time out.

• Wash the vegetables before feeding: Extract core fields from tool-returned data with a few lines of code; strip redundant HTML tags and extra JSON fields before feeding to the model.

• Pass through disconnect signals: Use AbortController or the framework's cancellation context. When the frontend disconnects, immediately and cleanly cancel the model task, freeing precious concurrency resources for the next living person.

Ultimate Lifesaver: Too Slow? Switch to Async Queueing!

If your Agent is stuck with heavy tasks like deep research, writing code, or long-form writing, and single execution will definitely exceed 30 seconds, stop trying to force it with regular HTTP requests.

Immediately switch to async task flow pattern: User clicks submit -> Backend pushes to a message queue (like Redis Stream or BullMQ) -> Instantly returns a Task ID to frontend -> Background distributed worker cluster processes slowly -> Frontend subscribes via SSE with the Task ID for typewriter effects.

This way, even if 10,000 people flood in concurrently, everyone queues nicely, and no one gets a 504 timeout package.