跪拜 Guibai
← All articles
Backend · AI Programming

RAG endpoint burns 8 CPU cores while chat uses 0.16: reranker is the real cost

By AI入职记 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

For teams containerizing RAG services, this pinpoints reranking as the dominant CPU cost and shows that a lexical fallback can cut compute by 12x. The measurement methodology lesson (warmup, repeated sampling, sufficient sample size) applies to any side-by-side benchmark of inference endpoints.

Summary

A Dockerized Spring Boot service exposes two endpoints: /api/chat, which forwards to DeepSeek, and /api/rag/ask, which runs embeddings and a cross-encoder reranker. Load tests with 50 requests show chat scales linearly with concurrency (QPS doubles, latency flat) because the bottleneck is the external model's network round-trip. RAG, on the other hand, peaks at 797-866% CPU (about 8 cores on a 10-core machine), and nearly all of that goes to the bge-reranker-base model running under ONNX Runtime's default CPU execution provider.

Replacing the cross-encoder with a purely lexical reranker (bigram matching plus weighted vector scores) drops CPU to 66-74%, a 12x reduction. Document embeddings are computed once at startup, so per-request embedding cost is tiny. The cross-encoder re-ranks 5 candidates per query, truncating each to 512 tokens, which saturates all 10 ONNX intra-op threads. The author also reveals a measurement trap: the first benchmark used only 6-12 requests and a single docker stats snapshot, leading to a false conclusion that embedding itself cost 4.9 cores. Reproducing with warmup and 50 requests corrected the number to 0.7 cores.

Practical deployment details include a multi-stage Dockerfile that caches Maven dependencies, an exec-form entrypoint so SIGTERM reaches the JVM, a named volume for model caches (including the DJL cache that defaults to ~/.djl.ai), and a compose file that binds only to 127.0.0.1. The .env file is auto-injected by compose and can override defaults silently. ONNX Runtime's default intra-op thread count equals the host core count, which can hurt small containers; the author leaves tuning that parameter as an open question.

Takeaways
— A Spring Boot RAG endpoint with a cross-encoder reranker peaked at 797-866% CPU (about 8 cores) under load, while the chat endpoint peaked at 11-16%.
— Switching from a cross-encoder reranker (bge-reranker-base, 266 MB) to a lexical reranker (LexicalReranker) reduced CPU peak from about 8 cores to about 0.7 cores, a 12x drop.
— Document embeddings are computed once at startup, so per-request embedding cost is only about 2.88 ms for a short query; the reranker is the real CPU hog.
— ONNX Runtime defaults to one intra-op thread per CPU core, so a single RAG request on a 10-core machine can consume all 8 cores; small containers may need explicit thread limiting via setIntraOpNumThreads.
— The first benchmark used 6-12 requests and a single docker stats snapshot, falsely reporting 492% CPU for lexical mode; after warming up and sampling over 1-2 seconds, the true value was 66-74%.
— Warmup is mandatory before load testing: the first request initializes the ONNX session and spiked CPU to 311% in a reproduction.
— The DJL model cache (~200 MB) does not live under Spring AI's model cache directory; set DJL_CACHE_DIR to put it on the mounted volume, otherwise it re-downloads on every container rebuild.
— The compose file auto-injects .env variables, so RAG_RERANK=cross in .env overrides the compose default of none; explicit override is needed to switch modes.
— Port binding is layered: app listens on 127.0.0.1 inside the container, Dockerfile overrides with SERVER_ADDRESS=0.0.0.0, and compose publishes only to host 127.0.0.1, keeping the service local.
— The service uses graceful shutdown with a 30s timeout; the Dockerfile's exec-form entrypoint ensures the JVM receives SIGTERM instead of the shell swallowing it.
Conclusions

The 12x CPU gap between cross-encoder and lexical reranking is a strong argument for evaluating whether semantic reranking is worth its compute cost in containerized RAG, especially when downstream generation latency already dominates end-to-end response time.

The author's initial measurement mistake is a lesson for anyone benchmarking inference: a single snapshot of docker stats can land on a startup spike or a steady-state dip, and without warmup, the first request's ONNX initialization inflates CPU readings dramatically.

The claim that 'reranker is the CPU bottleneck' is explicitly scoped to a CPU-only ONNX Runtime on Apple Silicon with a 512-token rerank window; changing any of those variables (GPU provider, smaller model, shorter window) could shift the bottleneck entirely.

ONNX Runtime's default thread-per-core behavior means a high-core-count host can monopolize all cores for a single request, which is a trap for people deploying to small containers without tuning intra-op threads.

The .env auto-injection gotcha is a subtle deployment hazard: Compose's environment merging can silently override your intended defaults, leading to unexpectedly loading a 266 MB reranker.

Concepts & terms
Reranker
In retrieval-augmented generation (RAG), a reranker is a model that re-scores the top-k retrieved chunks to improve relevance. It runs after initial retrieval and before sending context to the LLM. Cross-encoder rerankers process each candidate pair (query, chunk) jointly, which is accurate but computationally expensive; lexical rerankers use string matching or bag-of-words heuristics that are much cheaper.
ONNX Runtime CPUExecutionProvider
The CPU execution provider is the default backend for ONNX Runtime, running neural network inference on the host CPU rather than a GPU accelerator. It uses intra-op thread pools that default to the number of physical cores, which can saturate a container's available CPU if not limited.
Intra-op threads
ONNX Runtime's parallelism inside a single operator (e.g., a matrix multiply). The default thread count equals the number of CPU cores, so a single inference can use all cores, causing contention in small containers.
From the discussion
Featured comments
组合搭配函数 1 likes

The same endpoint measured a 7x difference, and the root cause is the measurement method, which is a hard truth to swallow. Averaging over a 5-minute window is a sound approach; the load test data finally adds up.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗