Backend · Interview · Java

LLM, RAG, Agent, MCP in One Request: The Interview Test That Trips Up Most Java Engineers

By Rain的Java大神之路 · Nov 4, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Enterprise AI work now runs through exactly this stack, and the failure modes are concrete and costly: unauthenticated MCP endpoints, retrieval misses near 40%, context-budget blowups, and agent loops that burn tokens. The Java-specific answers here, from dynamic tool provisioning to multi-tenant pre-filtering, are what separate demo agents from systems that survive production traffic.

Summary

One customer-service question, "why hasn't my order shipped," carries every concept in the modern AI stack: LLM as next-token generator, Embedding as semantic coordinates, cosine similarity as the ruler, vector databases and ANN indexes for scale, RAG with its bottleneck in retrieval rather than generation, tool calling where the model only names the function while Java code executes it, MCP as the standard socket that exposes legacy Spring Boot systems to AI, Skills as progressive-disclosure job manuals, and Agent as the loop that decides next steps under step and budget limits.

The material includes production Java: float storage with double accumulation to avoid precision loss, zero-vector short-circuits, SIMD dot products via the JDK Vector API, L2 normalization so cosine degrades to inner product, HNSW/IVF/PQ index choices with recall@K acceptance at 95-99%, RRF fusion for BM25 plus dense hybrid retrieval, a ReAct loop with four guards (max steps, error-to-text feedback, budget circuit breaker, fallback copy), and a three-minute Spring Boot to MCP Server conversion. The economics are spelled out: 28 tools inflate a single request from about 800 to 4200 tokens; converging to 5-8 tools drops it to about 1200. Semantic caching cuts inference cost 40-70%; model tiering cuts it another 40-60%.

A 16-row troubleshooting table spans hallucination, retrieval misses, context explosion, tool misuse, multi-tenant leakage, and idempotency failures. A 13-item interview Q&A compresses the decision rules: why cosine beats Euclidean for text, why RAG survives 1M-token contexts, and how MCP differs from function calling. The closing frame is that the job is putting uncertain model capability inside a deterministic engineering system, which is exactly what Java engineers already do; the same skeleton adapts to travel reimbursement assistants and ops alerting.

Takeaways

— RAG's bottleneck is retrieval, not generation; a naive pipeline fails at retrieval roughly 40% of the time.

— L2-normalizing vectors makes cosine similarity equal inner product, removing two square roots and a division from the query hot path.

— 1536-dim float vectors cost about 6KB per record; 1024 dims with PQ quantization is the common enterprise choice.

— ANN acceptance target is recall@K of 95-99%.

— 28 tools inflate a single request from about 800 to 4200 tokens; converging to 5-8 tools drops it to about 1200 and cuts mis-call rates.

— MCP HTTP endpoints default to unauthenticated JSON-RPC; any reachable client can list and call every tool.

— MCP spec v5 (2026-07-28) moves to a stateless core, removes Session ID, adds versioned extensions and W3C Trace Context.

— Semantic caching cuts inference cost 40-70%; model tiered routing cuts cost another 40-60%.

— Streaming responses drop first-token latency from roughly 8s to about 500ms.

— Changing the embedding model requires rebuilding the entire vector database.

— Indexing and querying must use the same embedding model, or similarity scores are pure noise.

— Production RAG combines advanced parsing, hybrid retrieval (BM25 + dense with RRF fusion), cross-encoder reranking, agentic routing, GraphRAG, and continuous evaluation via RAGAS or TruLens.

— Skills should load on demand through progressive disclosure, not be stuffed into the system prompt.

— SKILL.md should stay under 500 lines; reference files one level deep with a TOC for files over 100 lines.

— An agent loop needs a hard step limit, time/token budget circuit breaker, tool-call deduplication, and truncated tool results.

— Virtual threads absorb blocking retrieval IO without exhausting OS threads under high concurrency.

— Tenant and permission metadata should filter at the vector store retrieval stage, not after retrieval.

Conclusions

Treating an LLM as an unreliable third-party REST service reframes AI engineering as a reliability discipline, and it matches how Java shops already govern any external dependency.

The 800-to-4200 token blowup for 28 tools is the kind of concrete cost data that should drive tool-set design, yet most demo projects ignore it entirely.

The claim that MCP HTTP endpoints ship with no auth by default is a security fact worth verifying before exposing any tool server, given how easily a discovery endpoint becomes a call-everything endpoint.

MCP spec v5's stateless turn and the SSE deprecation mean teams that hard-code to one transport or one session model will pay a migration cost within a year.

Positioning Java engineers as "car builders, not engine builders" is a sharp career argument: model layers are commoditized, and the enterprise moat sits in governance, integration, and reliability.

Concepts & terms

RAG

Retrieval-Augmented Generation: retrieves relevant documents from an external knowledge base and inserts them into the prompt before generation, so the model answers from cited material instead of its frozen training data.

Embedding

A model that maps text to a high-dimensional vector of floats (768-3072 dims) so semantically similar texts land close together in vector space; the same model must be used for both indexing and querying.

Cosine similarity

A measure of the angle between two vectors, range [-1,1], used for text retrieval because it ignores vector length, which is influenced by text length; after L2 normalization it equals the inner product.

ANN index

Approximate Nearest Neighbor index: data structures like IVF, HNSW, and PQ that trade a small amount of recall for orders-of-magnitude speedups in similarity search over millions of vectors.

HNSW

Hierarchical Navigable Small World graph: the default ANN index in most vector databases, building a layered graph that jumps far at the top levels and descends greedily toward closer neighbors; best recall/latency balance, but memory-heavy and slow to build.

RRF

Reciprocal Rank Fusion: combines ranked lists from multiple retrieval sources by scoring each doc as the sum of 1/(k + rank), requiring no score normalization across sources.

Rerank (cross-encoder)

A second-stage model that takes the query and a document concatenated together to compute relevance, more accurate than vector similarity's two-tower approach but slower; applied to the top ~50 candidates to select the final 5-8.

Tool Calling

When a model outputs a structured request to invoke a named function with JSON arguments, and the application executes it; the model names the tool, the code performs the action, then the result feeds back into the conversation.

MCP

Model Context Protocol: an open standard from Anthropic (late 2024) that unifies how AI applications discover and call external tools, analogous to HTTP for web APIs; uses JSON-RPC 2.0 with Tools, Resources, and Prompts primitives.

Skills

A file-based convention (SKILL.md plus scripts and reference material) that captures how to perform a class of tasks; loaded on demand via progressive disclosure so many skills can exist without inflating the context window.

ReAct

Reasoning + Acting loop: the agent iterates Thought → Action → Observation until it converges or hits limits; the inner reasoning layer of agent systems.

Recall@K

The overlap between an ANN search's top-K results and the exact brute-force top-K results; production systems tune it to 95-99%.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗