跪拜 Guibai
← Back to the summary

LLM, RAG, Agent, MCP in One Request: The Interview Test That Trips Up Most Java Engineers

Interviewer asks you to explain LLM, RAG, Agent, MCP as one coherent story, and 90% of Java engineers freeze on the spot

📌 Read this first (spend 60 seconds)

Straight talk: LLM, RAG, Agent, MCP, vectors, cosine similarity… you've heard every one of these terms, but when an interviewer asks you to “explain them together coherently,” do you freeze on the spot?

Don't panic. That's precisely the signal that what you lack isn't knowledge points, but a big-picture map.

This article doesn't pile on concepts or recite definitions. It uses one real e-commerce customer service request as the main thread, walking you from “user asks: why hasn't my order shipped” all the way to the final streamed output. The technology used at every step, the Java code written, and the pitfalls hit are all laid on the table.

What you can take away from reading this?

Who is this for? Backend developers pivoting to AI, Java engineers building enterprise knowledge bases or smart customer service, and friends with interviews next week who are currently panicking.

Who is this not for? Anyone wanting to learn how to train large models or tune LoRA — not a single gradient here. This article only covers one thing: how to put uncertain model capability into a deterministic engineering system.

⏱️ Full text ~15,000 characters. Save it first, then read carefully. You can't finish it on your commute.

There's a small easter egg at the end: the article asks “what's the most painful pitfall you've hit in RAG or Agent rollout?” Chat a bit in the comments; I'll reply to each one. If you find it useful, hit like and share, don't let it collect dust in your bookmarks 👇

One request threading LLM, RAG, Agent, MCP: the full picture of Java engineers doing AI engineering

An interview high-frequency question: explain LLM, Agent, Skills, MCP, RAG, vectors, and cosine similarity as one coherent story. Most people can recite each concept individually; the moment they have to connect them, they freeze. This article uses one real customer service request as the main thread, combined with rollout experience in e-commerce customer service and enterprise knowledge base projects, to explain the principles, Java implementations, and pitfalls of each concept. It's for backend developers pivoting to AI. Save it first, then read carefully.

1. Set the big picture first: one sentence + two diagrams connect all concepts

The biggest pain point in learning AI concepts is “heard every one, can't connect them.” Don't charge into details; spend three minutes building global awareness first, and no matter how you study later, you won't get lost.

First, put all concepts into one metaphor system with a single sentence:

LLM is the engine; Embedding is the coordinate system of language; cosine similarity is the ruler; the vector database is the library's index card; RAG is the model's “external reference room”; Tool is the model's hands and feet; MCP is the unified USB socket; Skills are the job behavior manual; Agent is the digital employee who breaks down tasks himself, finishes work, and reviews afterward.

And the value of a Java engineer has never been “building the engine,” but building the whole car, repairing the highway, and building gas stations.

Then a “human language” quick reference table:

Concept One-sentence plain talk Analogy Java-side handle
LLM A text-continuation machine that only predicts the next token Brain / engine Spring AI, LangChain4j, WebClient calling API + SSE streaming
Embedding Turns text into semantic coordinates Coordinate system EmbeddingModel outputs float[], stored in pgvector / Milvus
Cosine similarity Looks at whether two vectors' directions are alike Ruler / compass cos = A·B / (|A||B|), used for TopK recall
RAG Open-book exam: look up material first, then answer External reference room chunk → vectorize → retrieve → assemble prompt
Skills Standard operating procedure for a class of tasks Job manual / toolbox @Tool methods + on-demand-loaded SKILL.md
MCP Unified protocol for tool integration USB-C socket spring-ai-starter-mcp-server-*
Agent Breaks down tasks, calls tools, loops on decisions Digital employee AiServices + ReAct loop / state machine

1.1 Five-layer architecture: figure out whose job each layer is

Putting concepts into a layered architecture makes responsibilities clear at a glance — same thinking as the MVC layering we're used to:

image.png

Layer One-sentence responsibility Java-side handle
① Model layer Provides “understanding and generation” compute HTTP/SSE calls, Spring AI ChatModel
② Context layer Turns semantics into computable distance MilvusVectorStore, PgVectorStore, ES 8.x dense_vector
③ Capability layer Lets the model see knowledge and touch systems LangChain4j ContentRetriever, Spring AI ToolCallback
④ Orchestration layer Lets the model decide what to do next itself AiServices, Spring StateMachine, ToolProvider
⑤ Engineering governance layer Gets AI stable, safe, and cheap into production WebFlux + SSE, Redis token bucket, Micrometer, gateway

💡 Technical highlight (interview bonus line): of these five layers, ①② are bought, ③④ are given by frameworks, and ⑤ is the enterprise's own moat. That one line basically nails “why hire Java rather than algorithm engineers.”

1.2 Concept dependency: how they mesh

image.png

One sentence to string them together: vectors and cosine similarity are the foundation of semantic computation, RAG installs a knowledge base for the LLM, Skills and MCP connect business hands and feet for the LLM, and Agent assembles all the capabilities into an autonomous agent that can complete tasks.

Next we break each down from the bottom up. Start with the unglamorous but decisive foundation.

2. LLM: don't deify it, it's a “next token predictor”

Many people treat large models as encyclopedias; that's the root of all engineering problems. Nail down three key recognitions first:

Add two more: it has no memory — so-called multi-turn conversation works only because we stuff the message history into the messages array again and again; it doesn't access the network or databases — strong in language understanding and generation, weak in precise calculation, real-time data, and private-domain knowledge.

With that understood, the position of a Java developer is clear:

💡 One-line memory: in the Java world, a large model is essentially just another third-party REST service that can chat — except it's extraordinarily slow, expensive, and occasionally talks nonsense. Our job isn't to switch to algorithm engineering but to be an LLM Application Engineer: govern it like any “unreliable dependency” — timeouts, retries, circuit breakers, degradation, observability.

2.1 Token, context, temperature: all budgets

Concept Essence Engineering meaning
Token The model's smallest billing and processing unit; subwords, not characters Chinese 1 char ≈ 11.5 tokens (varies by tokenizer); cost and latency both billed per token
Prompt The input template sent to the model Prompt is also “code”: must be versioned, evaluated, regression-tested
Context window Everything the model can see in one pass A budget, not capacity: how much retrieval to stuff, how much history to keep, how much output to reserve — all must be accounted for
Temperature Controls the “sharpness” of the probability distribution Extraction/classification/factual QA at 0~0.3, creative writing 0.7+
Structured output Makes the model emit valid JSON that maps to Java types BeanOutputConverter + validation + retry on failure

Longer context costs more and increasingly suffers “Lost in the Middle.” So don't treat the context as a hard drive; it's a scarce resource — a point that returns when we discuss Skills later.

2.2 Don't treat the LLM as a database

A division-of-labor principle summarized after hitting pitfalls; extremely useful in real development:

Factual data: query the database / knowledge base; process operations: go through business APIs / MCP; language expression: only then hand it to the LLM.

LLMs are good at reasoning and organizing expression, weak at long-term memory and precise facts. Making it do what it's bad at is the start of hallucination.

The model has all these flaws — how to patch them? Give it external attachments: first attach knowledge (RAG), then attach hands and feet (Tool/MCP). But before that, one mathematical problem must be solved: how does a machine understand “meaning”?

3. Vectors + cosine similarity: turning “semantics” into computable numbers

3.1 Why vectors are needed

Traditional SQL only excels at exact matching and fuzzy matching; like '%car%' will never match automobile. Keyword retrieval matches characters; we want to match meaning.

Embedding models compress a piece of text into a string of high-dimensional floats (commonly 768 / 1024 / 1536 / 3072 dims). Semantically closer texts land closer together in vector space:

"如何申请年假"  → [0.012, -0.231,  0.548, ...]  ┐ small angle → semantically near-equivalent
"年假怎么请"    → [0.009, -0.198,  0.561, ...]  ┘
"如何申请离职"  → [-0.34,  0.087, -0.112, ...]  → large angle → semantically different

Several key engineering recognitions, all bought with blood and tears:

How to choose dimensions? Higher dimensions express semantics more finely, but storage and compute cost grow linearly. At float storage, 1536 dims × 4B ≈ 6KB/record. Enterprise-grade common choice: 1024 dims with PQ quantization compression.

3.2 Cosine similarity: why not Euclidean distance

The formula is simple, range [-1, 1]:

cos(θ) = (A·B) / (‖A‖ × ‖B‖) = Σ(aᵢbᵢ) / (√Σaᵢ² × √Σbᵢ²)

Know how to read the score, have a scale in your head:

Similarity Meaning Scenario
1.0 Exactly same direction synonymous sentences
0.7+ Highly related common recall threshold
0.0 Orthogonal, unrelated —
negative Opposite direction rare in text vectors

Why does text default to cosine rather than Euclidean?

Metric Formula Range Fit
Cosine similarity Σ(aᵢbᵢ) / (‖a‖‖b‖) [-1, 1] default for text semantic retrieval (direction only, ignore length)
Inner/dot product Σ(aᵢbᵢ) (-∞, +∞) recommendation systems (magnitude encodes popularity, confidence)
Euclidean distance √Σ(aᵢ-bᵢ)² [0, +∞) image features and cases where magnitude itself is meaningful

The magnitude of a text vector is influenced by text length — “today is nice weather” and “today is nice weather nice weather nice weather” mean the same thing, but Euclidean distance grows. Cosine only looks at direction and is insensitive to length. Also, in high-dimensional space Euclidean suffers the curse of dimensionality: distances converge and discriminative power deteriorates.

3.3 Three mathematical relations: fail to answer these and you're exposed

⭐ Must-answer in interviews:

  1. After normalization, cosine == inner product. Do L2 normalization so ‖a‖=‖b‖=1; substitute into the formula and get cosθ = a·b. Engineering value: at query time only one multiply-accumulate is needed, saving a square root and a division every time. In FAISS, IndexFlatIP with normalized vectors computes exactly cosine.
  2. After normalization, larger cosine ⇔ smaller L2 distance. ‖a-b‖² = 2 - 2cosθ = 2(1-cosθ). The ranking results of all three metrics are exactly equivalent, differing only in score scale.
  3. Ranking equivalence ≠ threshold portability. If you use the cosine score as an absolute threshold filter (say “below 0.6 means no answer”), the choice of metric and the normalization state matter critically — mixing normalized/unnormalized vectors makes thresholds fail outright.

3.4 Three pitfalls interviewers love to probe

Pitfall Phenomenon Essence and response
Similar ≠ relevant “Apple's CEO” and “Tim Cook” may be far apart in vector space Vectors measure distributional similarity of wording, not factual relation. Compensate with hybrid retrieval + Rerank + knowledge graph
Hubness hubs Some vectors become “universal neighbors,” ranked first for any query An inherent high-dimensional phenomenon. Mitigate by deduplication, cluster dispersion, diversity constraints (MMR)
Duplicate documents inflating scores Storing the same document 10 times makes it easier to recall Do content fingerprint deduplication at ingestion; aggregate by doc_id after recall

3.5 Production code: three details matter more than the formula

public final class CosineSimilarity {

    private CosineSimilarity() {}

    /**
     * Generic cosine similarity, three engineering details:
     * ① accumulate in double — 1536-dim float accumulation loses precision and pollutes Top-K ranking;
     * ② zero vectors must short-circuit — otherwise NaN pollutes the ranking, making Comparator behavior unpredictable;
     * ③ dimension check — indexing and query coordinate systems must match.
     */
    public static double cosine(float[] a, float[] b) {
        if (a.length != b.length) {
            throw new IllegalArgumentException("vector dimension mismatch: " + a.length + " vs " + b.length);
        }
        double dot = 0.0, na = 0.0, nb = 0.0;
        for (int i = 0; i < a.length; i++) {
            dot += (double) a[i] * b[i];   // promote to double before accumulation
            na  += (double) a[i] * a[i];
            nb  += (double) b[i] * b[i];
        }
        if (na == 0.0 || nb == 0.0) {      // zero-vector short-circuit, prevent NaN
            return 0.0;
        }
        return dot / (Math.sqrt(na) * Math.sqrt(nb));
    }

    /** L2-normalize before ingestion: subsequent retrieval degrades to inner product, saving 2 square roots + 1 division */
    public static float[] l2Normalize(float[] v) {
        double n = 0.0;
        for (float x : v) n += (double) x * x;
        if (n == 0.0) return v;
        double inv = 1.0 / Math.sqrt(n);
        float[] out = new float[v.length];
        for (int i = 0; i < v.length; i++) out[i] = (float) (v[i] * inv);
        return out;
    }

    /** inner product between normalized vectors == cosine similarity (use this on the retrieval hot path) */
    public static double dot(float[] a, float[] b) {
        double s = 0.0;
        for (int i = 0; i < a.length; i++) {
            s += (double) a[i] * b[i];
        }
        return s;
    }
}

💡 Technical highlight: float storage halves memory and bandwidth, but accumulate in double to prevent precision loss; zero vectors return 0 instead of NaN — NaN makes Comparator ranking unpredictable, a real pitfall. And the trick “normalize at write, degrade to dot product at query” measured 2~3x scoring performance gains on million-scale vectors, and roughly 40% on ten-million-scale batch retrieval.

Batch dot products can also be handed to SIMD:

import jdk.incubator.vector.FloatVector;
import jdk.incubator.vector.VectorOperators;
import jdk.incubator.vector.VectorSpecies;

/**
 * JDK Vector API (jdk.incubator.vector, incubating, requires --add-modules):
 * lets the JIT emit SIMD instructions; batch dot product reaches multiples of scalar-loop performance.
 * Positioning: custom operators for offline index build/rerank stages; online retrieval always goes to a vector database.
 */
public final class SimdDotProduct {

    private static final VectorSpecies<Float> SPECIES = FloatVector.SPECIES_PREFERRED;

    public static double dot(float[] a, float[] b) {
        int i = 0;
        int upperBound = SPECIES.loopBound(a.length);
        FloatVector acc = FloatVector.zero(SPECIES);

        for (; i < upperBound; i += SPECIES.length()) {
            FloatVector va = FloatVector.fromArray(SPECIES, a, i);
            FloatVector vb = FloatVector.fromArray(SPECIES, b, i);
            acc = acc.add(va.mul(vb));
        }

        double sum = acc.reduceLanes(VectorOperators.ADD);
        for (; i < a.length; i++) {         // tail shorter than one vector width, scalar fallback
            sum += (double) a[i] * b[i];
        }
        return sum;
    }
}

With vectors and a ruler in hand, the next question is practical: once data volume grows, are we really going to for-loop through every entry?

4. Vector databases and ANN indexes: why you can't just for-loop

First do the math: 1 million 1024-dim vectors; each query needs 1024 multiply-adds, roughly 2 billion float operations per query; just reading the 4GB of vectors from memory is enough trouble. 100 concurrent? The CPU queues until morning.

The way out: allow yourself to “not compute quite so exactly” — ANN (approximate nearest neighbor) trades a little recall for orders of magnitude in speed.

Index Core idea Pros Cost Fit
Flat brute-force full comparison 100% recall, no training O(N×dim), slow < 100k entries, or as recall baseline
IVF K-means clustering into buckets, search only nprobe nearest clusters memory-friendly, scalable boundary vectors easily missed 10M+, accepts offline rebuild
HNSW hierarchical navigable small-world graph best recall/latency overall, mainstream default high memory, slow build, deletion-unfriendly general-purpose first choice
PQ product quantization compresses vectors memory down an order of magnitude precision loss, needs rerank compensation 100M scale, memory-constrained

One sentence to understand HNSW: skip list solves “jump far”; small-world graph solves “jump accurately” — leap from high-level entry points, descend layer by layer, greedily walk toward closer neighbors at each level, approaching the target with an O(log N) access path.

In production, accept with recall@K (the Top-K overlap between ANN results and brute-force results), generally tuned to 95%~99%, while latency drops from seconds to milliseconds.

4.1 Java-side selection table

Scenario Choice Reason
Already have an ES cluster Elasticsearch 8.x dense_vector one system eats both BM25 + vectors, hybrid retrieval easiest
Already have PostgreSQL pgvector transactions, permissions, backups all reused, zero new ops
10M~100M scale, needs low latency Milvus / Qdrant specialized vector DBs, HNSW/PQ parameters finely tunable
unit tests / small data in-memory Map + cosine don't bring in heavy components, SimpleVectorStore is enough

⚠️ Pitfall warning: don't implement your own ANN index. The highlight should be “how to choose an index, how to tune nprobe / efSearch, how to accept with recall@K,” not hand-writing HNSW. When retrieval latency exceeds 1s at ten-million scale, the standard solution is HNSW index + local cache of hot vectors (Caffeine) + hot/cold data separation.

In the pgvector scenario, push computation down into the database instead of pulling all vectors into Java memory:

-- <=> is pgvector's cosine distance operator; 1 - distance = cosine similarity
SELECT id, content, 1 - (embedding <=> :qvec) AS score
FROM doc_chunk
WHERE tenant_id = :tenantId
ORDER BY embedding <=> :qvec
LIMIT 5;

Wrap it in Spring Data JPA with a native query:

@Query(value = """
    SELECT id, content, 1 - (embedding <=> :queryVector) AS similarity
    FROM document_chunks
    WHERE 1 - (embedding <=> :queryVector) > :threshold
    ORDER BY similarity DESC
    LIMIT :topK
""", nativeQuery = true)
List<DocSimilarity> findTopKByCosine(@Param("queryVector") float[] queryVector,
                                     @Param("threshold") double threshold,
                                     @Param("topK") int topK);

Milvus declarative config — note the dimension must strictly align with the Embedding model:

@Configuration
public class MilvusVectorConfig {

    @Bean
    public VectorStore milvusVectorStore(EmbeddingModel embeddingModel) {
        return MilvusVectorStore.builder()
                .embeddingModel(embeddingModel)
                .host("milvus-internal-host")
                .port(19530)
                .collectionName("knowledge_base")
                .dimension(1536)                 // strictly aligned with Embedding model dimension
                .indexType(IndexType.HNSW)       // high-performance vector index
                .metricType(MetricType.COSINE)   // cosine metric
                .build();
    }
}

The foundation is done; we officially enter the application layer. First up, the interview's first gate — RAG.

5. RAG: open-book exam, but the bottleneck is retrieval, not generation

First correct a high-frequency wrong answer:

RAG is not “vector database + LLM”; it's a whole coordinated mechanism of “external knowledge management + retrieval system + generation model.” It solves four problems: stale knowledge, unknown private knowledge, easy fabrication (hallucination), untraceable answers.

5.1 The complete chain: two stages, must be explained in order

image.png

5.2 The high-score answer: first explain “why naive RAG fails”

Naive RAG failure mode Typical symptom Corresponding fix
Recall inaccuracy “Apple CEO” fails to recall “Tim Cook” hybrid retrieval (dense + BM25), query rewriting
semantics ≠ similarity searching for ID AG-2024-001 recalls docs full of “product ID” wording but wrong numbers BM25 exact match as fallback, weight by scenario
torn context fixed 512-char hard chunking severs arguments mid-way structure-aware chunking, parent-child blocks, semantic chunking
multi-hop reasoning failure “the QS ranking of the alma mater of X company's CEO” can't be found query decomposition, Agentic RAG multi-round retrieval
global questions collapse “what's the core theme of this batch of documents” GraphRAG / global summary; vector retrieval is naturally useless here
too much recall, lots of noise stuffing all Top-20 in, model gets distracted Rerank re-ranking, only feed top-5~8

Stacking the fixes gives the 2026 production-grade RAG formula:

Production RAG = advanced parsing (OCR / layout recovery) + hybrid retrieval (BM25 + dense, RRF fusion) + neural reranking (cross-encoder) + agentic routing (dynamic retrieval depth by question complexity) + GraphRAG (global/multi-hop questions) + continuous evaluation (RAGAS / TruLens, quantifying recall@k, faithfulness, hallucination rate)

📌 Here's a watershed-level recognition: RAG's bottleneck is retrieval, not generation. The naive pipeline fails with roughly 40% probability at the retrieval step; if retrieval goes in wrong, later reranking and generation optimization are useless.

And to answer a high-frequency doubt: context windows are at 1M now, do we still need RAG? Yes. Three reasons: cost and latency (full ingestion every request is unacceptable), attention decay (more stuffed in doesn't mean better remembered), real-time freshness and permission granularity (knowledge base updates per entry, filters per tenant). The correct posture: RAG handles precisely finding from the vast pool; long context handles careful reading and reasoning on what was found.

5.3 Five key design decisions

① Chunking: fixed-length hard cutting is the culprit of semantic severing. Practical parameters: 500~1000 tokens per chunk, 10%~20% overlap, structure-aware splitting by heading/paragraph, then parent-child blocks — child blocks for vector matching; upon a hit, the parent block's full content goes into the context. Headings and metadata should ride along for filtering at retrieval and provenance at answer time.

② Embedding consistency: query and doc must share the same model and dimension; already covered, no need to expand.

③ Hybrid retrieval + RRF fusion: pure vectors miss exact keywords (model number RTX-5090, regulation number HR-2025-013); BM25 complements exactly. RRF (Reciprocal Rank Fusion) scores as score = Σ 1/(k + rank), needing no normalization of two differently-scaled scores; far more stable than weighted summation.

④ Reranking: vector recall uses a “two-tower model,” query and doc independently encoded, unable to interact deeply; Rerank is cross-encoding, concatenating Q+D to compute relevance — accurate but slow, so it only re-ranks the coarse top 50 and takes top-58.

⑤ Generation constraints: hardcode in the prompt “answer only from provided material; say so if unknown” + force citations. That's the first gate against hallucination.

5.4 Core code: a RAG loop fit for production

In Spring AI, RAG can be woven in with one Advisor; business code barely notices:

@Configuration
public class RagConfig {

    /** In production swap to PgVectorStore / MilvusVectorStore / ElasticsearchVectorStore */
    @Bean
    VectorStore vectorStore(EmbeddingModel embeddingModel) {
        return SimpleVectorStore.builder(embeddingModel).build();
    }

    /**
     * Three key RAG constraints:
     * ① topK + similarityThreshold: control the context budget, rather less than too much;
     * ② system prompt hardcodes “answer only from provided material, say so if unknown” — first gate against hallucination;
     * ③ bind citation sources for traceability (enterprise compliance requirement).
     */
    @Bean
    ChatClient chatClient(ChatClient.Builder builder, VectorStore vectorStore) {
        return builder
                .defaultSystem("""
                        You are an enterprise knowledge assistant. Only answer based on the 【retrieved material】,
                        and annotate the basis at the end with [source: filename#chunk-id].
                        When the material lacks the information, answer directly 「not covered in material」, no speculation.
                        """)
                .defaultAdvisors(
                        QuestionAnswerAdvisor.builder(vectorStore)
                                .searchRequest(SearchRequest.builder()
                                        .topK(8)
                                        .similarityThreshold(0.6)
                                        .build())
                                .build())
                .build();
    }
}

// Call side: retrieval woven in by the Advisor; business code barely sees RAG
@RestController
public class KnowledgeController {

    private final ChatClient chatClient;

    public KnowledgeController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @GetMapping("/kb/ask")
    public String ask(@RequestParam String q) {
        return chatClient.prompt().user(q).call().content();
    }
}

📎 Note: Spring AI's APIs vary slightly across minor versions (e.g., SimpleVectorStore.builder vs older constructors); follow the official docs for your version.

Going one step further, two-stage retrieval + streaming output:

@Service
@RequiredArgsConstructor
public class KnowledgeQAService {

    private final VectorStore vectorStore;
    private final RerankClient rerankClient;
    private final ChatModel chatModel;

    /** Two-stage retrieval: expand coarse recall → rerank to Top-5, each with its own job */
    public List<Document> retrieve(String query) {
        List<Document> candidates = vectorStore.similaritySearch(
                SearchRequest.builder()
                        .query(query)
                        .topK(20)                      // expand recall, leave room for Rerank
                        .similarityThreshold(0.6)      // threshold filter, don't feed noise to the model
                        .build());
        return rerankClient.rerank(query, candidates, 5);
    }

    /** Streaming Q&A: first-token latency from 3s down to ~300ms scale */
    public Flux<String> streamAnswer(String query) {
        List<Document> docs = retrieve(query);
        String context = docs.stream()
                .map(d -> "[%s] %s".formatted(d.getId(), d.getText()))
                .collect(Collectors.joining("\n---\n"));

        String systemPrompt = """
                You are an enterprise knowledge base assistant. Rules:
                1. Only answer based on the 【material】; if insufficient, you must answer "no relevant information"
                2. Annotate each conclusion with source id [DOC-xxx]
                3. Forbidden to fabricate processes, names, amounts

                【Material】
                %s
                """.formatted(context);

        return chatModel.stream(new Prompt(
                new SystemMessage(systemPrompt),
                new UserMessage(query)))
            .mapNotNull(r -> r.getResult().getOutput().getText());
    }
}

When there are multiple retrieval sources, use virtual threads for parallel recall + RRF fusion:

@Service
public class HybridRetriever {

    private final VectorStore vectorStore;   // Spring AI abstraction: PGVector / Milvus / Redis
    private final Bm25Searcher bm25;
    private final ExecutorService vt = Executors.newVirtualThreadPerTaskExecutor();

    public List<Doc> retrieve(String query, int topK) {
        // parallel recall on two paths: virtual threads let blocking IO not occupy OS threads; ten-thousand concurrency won't blow the pool
        CompletableFuture<List<Doc>> semantic =
                CompletableFuture.supplyAsync(() -> vectorStore.similaritySearch(
                        SearchRequest.builder()
                                .query(query)
                                .topK(topK * 3)          // recall more for Rerank
                                .similarityThreshold(0.6)
                                .filterExpression("tenantId == '" + TenantContext.get() + "'") // permission pushdown
                                .build())
                        .stream().map(Doc::from).toList(), vt);

        CompletableFuture<List<Doc>> lexical =
                CompletableFuture.supplyAsync(() -> bm25.search(query, topK * 3), vt);

        return rrf(List.of(semantic.join(), lexical.join()), 60).stream()
                .limit(topK).toList();
    }

    /** RRF: score = Σ 1/(k + rank); no score normalization, comparable across sources */
    private List<Doc> rrf(List<List<Doc>> ranks, int k) {
        Map<String, Double> score = new HashMap<>();
        Map<String, Doc> index = new HashMap<>();
        for (List<Doc> rank : ranks) {
            for (int i = 0; i < rank.size(); i++) {
                Doc d = rank.get(i);
                index.putIfAbsent(d.id(), d);
                score.merge(d.id(), 1.0 / (k + i + 1), Double::sum);
            }
        }
        return score.entrySet().stream()
                .sorted(Map.Entry.<String, Double>comparingByValue().reversed())
                .map(e -> index.get(e.getKey()))
                .toList();
    }
}

💡 Technical highlight: metadata filtering pushed down to the vector store, searching within the tenant/permission partition first, avoiding “retrieved but not authorized to see” privilege leakage; virtual threads handle blocking IO so you don't have to rebuild the whole chain in WebFlux to sustain high concurrency.

5.5 Ingestion side: chunking, async vectorization, caching

@Service
public class IngestService {

    /** offline ingestion: chunk with overlap, preventing semantic severing */
    public void ingest(List<Document> docs) {
        List<Document> chunks = new TokenTextSplitter(
                800, 200, 80, 5000, true).apply(docs);  // chunk 800 tokens, overlap 200
        vectorStore.add(chunks);                        // internally calls EmbeddingModel for vectorization
    }
}

@Service
public class EmbeddingService {

    /** async vectorization + MD5 cache: avoids calling Embedding API for identical text */
    @Async
    public CompletableFuture<List<Float>> embedAsync(String text) {
        String cacheKey = DigestUtils.md5Hex(text);
        List<Float> cached = cacheService.get(cacheKey);
        if (cached != null) {
            return CompletableFuture.completedFuture(cached);
        }
        List<Float> vector = embeddingClient.embed(text).getResult().getOutput();
        cacheService.put(cacheKey, vector, Duration.ofHours(24));
        return CompletableFuture.completedFuture(vector);
    }
}

💡 Technical highlight: async non-blocking doesn't stall the main thread; MD5 cache hits skip the API call, saving 30%+ of Embedding costs in production.

5.6 Two unavoidable ops problems

The knowledge base updated, but the vector DB is still stale — what to do? Event-driven + version comparison + transactional update:

image.png

How do you prove your RAG got better? Build a golden eval set (100~500 Q&A pairs), track three metrics: Recall@K (did it retrieve), Faithfulness (did it fabricate), answer relevance. Every chunking/model/threshold change reruns regression, wired into CI as a gate. RAG without automated evaluation is faith — data speaks, not feelings.

Knowledge problems are solved, but users don't just ask questions — they also want to check orders, file tickets, start refunds. That's where tools come in.

6. Tool Calling: the model doesn't execute tools, it only “names them”

The concept most often answered wrong:

❌ Wrong understanding: the AI calls the interface itself. ✅ Right understanding: the AI says “I need to call queryOrderStatus, parameter orderId=SO2026…,” the real execution is our Java program, which runs it and feeds the result back to the model, and the model then composes language.

user asks → AI analyzes intent → AI outputs “which tool + parameter JSON”
        → our program executes (with auth, rate limiting, audit) → result fed back to AI
        → AI generates the final answer from the result

This “misalignment” is precisely the security boundary that Java engineers hold: every real business action lives in your code — auth, idempotency, audit, canary — none of it can run away.

6.1 @Tool: the description is the API doc the model sees

@Component
public class OrderTools {

    private final OrderService orderService;

    /** The tool description is the API doc the model sees; write it unclearly and the model will mis-call */
    @Tool("Query order status and logistics tracking by order number. Read-only operation, does not modify any data.")
    public String queryOrderStatus(
            @P("order number, e.g. SO202610150001") String orderId) {
        // strict input validation: return explicit error text on invalid format so the model can self-correct
        if (!orderId.matches("SO\\d{8,}")) {
            return "Invalid order number format; should start with SO followed by digits, e.g. SO202610150001";
        }
        return JsonUtils.toJson(orderService.queryStatus(orderId));
    }
}

Annotation-style declaration is zero-intrusion; the framework handles argument parsing, type conversion, and result serialization automatically. Note the defensive-validation trick: on invalid parameters, don't just throw an exception; return an error message the model understands, and it will correct the parameter and retry.

Tool call failures and mis-filled parameters are the norm; production needs a retry feedback loop:

public ToolResult executeWithRetry(ToolCall call, int maxRetries) {
    for (int i = 0; i < maxRetries; i++) {
        try {
            Tool tool = toolRegistry.get(call.name());
            if (tool == null) throw new ToolNotFoundException("tool does not exist: " + call.name());
            validateParams(tool.getInputSchema(), call.arguments());  // JSON Schema strict validation
            return tool.execute(call.arguments());
        } catch (ToolValidationException e) {
            call.setFeedback("parameter error: " + e.getMessage());          // feedback to model for self-correction
        }
    }
    return new ToolResult("error", "still failing after multiple retries, please ask in a different way");
}

6.2 Runtime convergence of the tool set: an optimization worth its price

Why can't all @Tools be attached to the model at once? Three reasons:

🚀 Technical highlight: use ToolProvider to dynamically deliver the tool set at runtime per tenant/role/intent. After converging the tool set from 28 to 5~8, single-request tokens dropped from 4200 to 1200, and mis-call rates dropped significantly.

@Bean
ToolProvider scopedToolProvider(OrderTools orderTools, RefundTools refundTools) {
    return (ToolProviderRequest request) -> {
        UserContext ctx = UserContextHolder.current();   // current tenant/role, passed through the gateway
        ToolProviderResult.Builder builder = ToolProviderResult.builder();

        if (ctx.hasPermission("order:read") && looksLikeOrderQuestion(request.userMessage())) {
            builder.add(toolSpecification("queryOrderStatus"), executor(orderTools));
        }
        if (ctx.hasPermission("order:refund")) {         // high-risk write operation separately authorized, not given by default
            builder.add(toolSpecification("applyRefund"), executor(refundTools));
        }
        return builder.build();
    };
}

📎 Note: toolSpecification / executor are illustrative; in practice use LangChain4j's ToolSpecifications utility classes; method signatures vary by version, follow the official docs.

Tools are “hands and feet”; who manages “what process this type of job should follow”? That's Skills.

7. Skills: don't stuff the “operating manual” into System Prompt

Skills are how “how a class of tasks should be done” is sedimented into files (SKILL.md + scripts + reference material), a behavior manual the Agent loads only when needed.

7.1 Soul design: progressive disclosure

image.png

Why is this design worth money? Compare:

Comparison Stuffed in System Prompt Skills
Load timing always resident, every character eats budget loaded on demand, unused means zero tokens
Scale ceiling a dozen flows blow up the context 100 Skills cost only summary overhead
Maintainability one change breaks everything, can't version one folder one responsibility, independently evolvable

Using a Java analogy: Skills are to Agent what “design pattern manual + operating procedure” is to a new employee. You wouldn't make a new hire memorize 500 pages of SOP on day one — you'd give them a checklist saying “when you hit scenario XX, go read chapter 3.” Conversely, the System Prompt is the “job description,” which must be resident but must stay extremely short.

7.2 Three hard rules for landing Skills

  1. SKILL.md body kept within 500 lines; split if longer;
  2. Reference files only one level deep — nested too deep, the Agent reads only part of the content and gets truncated information;
  3. Reference files over 100 lines must have a table of contents (TOC) at the top, letting the Agent see the whole picture even on partial reads.

A skill definition looks like this:

# skill-definition.yaml —— a “ticket classification” skill
name: ticket-classifier
version: 1.2.0
description: automatic user ticket classification and priority determination
prompt_template: prompts/classify.st        # prompt template
tools:                                      # dependent tools (exposed via MCP)
  - query_user_profile
  - lookup_sla_policy
inputs:  [ticket_content, user_level]
outputs: [category, priority, reason]
eval_dataset: eval/ticket-golden-500.jsonl  # regression eval set

7.3 Three high-frequency confusions, cleared once

📎 Trend note: Agent Skills has become an open de-facto standard — the same folder structure is directly readable by Claude Code, Cursor, GitHub Copilot, Codex CLI, and Gemini CLI; “write once, run everywhere.” JVM-side abstractions now exist too (e.g., LangChain4j's dev.langchain4j.skills); the pattern is moving from “document convention” to “framework capability.”

Skills manage “how,” MCP manages “what you touch,” and MCP deserves its own big chapter — it's the most noteworthy new protocol in the Java world in the last two years.

8. MCP: the “USB-C interface” of the AI world

MCP (Model Context Protocol) is an open standard open-sourced by Anthropic in late 2024, unifying “how AI applications access external data and tools safely and standardly.” Analogy: HTTP unified Web API discovery and calling; MCP unified AI tool discovery (tools/list) and calling (tools/call).

8.1 It solves the M×N integration disaster

Before MCP: OpenAI Function Calling is one set, Claude's private format another, LangChain yet another — every tool needs N adapter sets. After MCP: implement the standard protocol and any MCP-supporting client can call you.

image.png

8.2 Three core primitives (based on JSON-RPC 2.0)

Primitive Controlled by Meaning Example
Tools the model executable functions the model can call query order, write file
Resources the application context data attached for the model file contents, config, Git history
Prompts the user predefined prompt templates slash commands, menu items

There is also Sampling: the Server reversely requests an LLM for reasoning, used for recursive intelligent tasks.

8.3 Transport evolution (timeliness knowledge, bonus in interviews)

Transport Scenario Status
STDIO local process, client starts the Server directly fits local CLI tools, can't cross network
SSE + HTTP remote deployment long-lived connections eat memory (~80KB/connection), weak-network disconnection rates high; marked deprecated since Spring AI 2.0
Streamable HTTP remote deployment, first choice request-response and streaming push folded into one HTTP semantics; gateways/load balancers/auth middleware can all intervene normally
Stateless Streamable HTTP stateless, Serverless memory < 5KB/request, requests routable to any instance

⏱️ 2026-07-28 new spec (v5, largest revision ever) three key changes:

  1. full turn from “stateful connection” to “stateless core” — protocol-level init handshake removed, Session ID removed, every request self-contained, horizontally scalable, Serverless-capable;
  2. versioned extension framework introduced, Extensions become first-class citizens (MCP Apps, Tasks), new server/discover capability discovery, TTL/cache semantics;
  3. production hardening: strengthened OAuth 2.0 / OIDC adaptation, full JSON Schema 2020-12, and W3C Trace Context tracing — call chains and evidence retention traceable, decisive for enterprise compliance.

8.4 MCP vs Function Calling: what's the relationship

Function Calling MCP
Positioning model capability: output structured call intent tool supply protocol: discovery, transport, auth
Coupling bound to a specific model vendor model-agnostic, switch models without touching tools
Reuse each app writes its own implement once, reused by many apps/models

One sentence: Function Calling is the verb, MCP is the USB-C interface. The former is the model-side “deciding which function to call,” the latter is the engineering-side “how functions/resources connect standardly”; complementary, not conflicting.

8.5 Why Java programmers must pay attention

Enterprise legacy core systems (orders, payments, risk control, ERP, supply chain) are overwhelmingly written in Java. MCP's value is exposing these legacy systems to AI with “zero business changes.”

Turn a Spring Boot service into an MCP Server in three minutes:

<!-- choose one: WebMVC project / WebFlux reactive project / local STDIO tool -->
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>
</dependency>

# application.yml
spring:
  ai:
    mcp:
      server:
        name: order-mcp-server
        version: 1.0.0
        protocol: STREAMABLE            # SSE deprecated since 2.0; options STREAMABLE / STATELESS
        streamable-http:
          mcp-endpoint: /api/mcp        # Streamable HTTP endpoint

@SpringBootApplication
public class OrderMcpServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(OrderMcpServerApplication.class, args);
    }

    /**
     * Existing Service unchanged a single line; wrap it and it becomes an AI-callable tool:
     * auto-configuration registers all ToolCallbacks as MCP Tools.
     */
    @Bean
    ToolCallbackProvider orderTools(OrderService orderService) {
        return MethodToolCallbackProvider.builder()
                .toolObjects(orderService)      // internally @Tool-annotated methods
                .build();
    }
}

On the client side, one ToolProvider can attach multiple MCP Servers:

// LangChain4j style: tool recall decided by the model runtime
@Bean
ToolProvider mcpToolProvider() {
    McpClient orderClient = new DefaultMcpClient.Builder()
            .transport(new HttpMcpTransport.Builder()
                    .sseUrl("http://order-mcp:8080/api/mcp")
                    .logRequests(true)
                    .logResponses(true)
                    .build())
            .build();

    return McpToolProvider.builder()
            .mcpClients(List.of(orderClient))
            .failIfOneServerFails(false)     // one Server down doesn't affect other tools
            .build();
}

New tools can also be discovered and registered dynamically at runtime — adding an external tool just means deploying an MCP Server; the app auto-discovers it, no restart, true hot-plug:

@Bean
public List<ToolCallback> dynamicMcpTools(McpClientManager manager) {
    return manager.getClients().stream()
            .flatMap(client -> client.listTools().stream())
            .map(tool -> (ToolCallback) tool)
            .collect(Collectors.toList());
}

8.6 Security red lines and write-operation discipline

⚠️ Security red line: the Spring AI official docs state clearly — an MCP Server over HTTP transport exposes by default an unauthenticated JSON-RPC endpoint; any client that can reach it can list and call every tool. A security boundary must go in front before go-live: OAuth2 / OIDC auth, gateway rate limiting, tool-level permissions, call auditing. Never expose an MCP Server naked on the intranet.

Write-operation tools also follow three disciplines: permission check, second confirmation (human-in-the-loop HITL), idempotency:

/** write-operation tool: permission + second confirmation + idempotency + audit, none can be missing */
@Tool("submit expense report, write operation, requires user confirmation")
public ExpenseResult submitExpense(SubmitExpenseRequest request) {
    // idempotency key control: model retries or network jitter cannot cause duplicate submission
    return idempotentService.execute(
            request.idempotentKey(),
            () -> expenseService.submit(request));
}

Supporting measures: idempotencyKey + business outbox + unique index; tool whitelist, read-only tools prioritized, high-risk write operations not open by default.

Hands and feet are complete; finally we need a “person who works on his own” to organize them — Agent takes the stage.

9. Agent: from “can think” to “can deliver”

Agent = LLM (brain) + Planning + Memory + Tools (hands and feet), the core being autonomously deciding what to do next. Unlike an LLM's “one question, one answer,” an Agent is “give it a goal, it completes it.”

9.1 The ReAct loop: think → act → observe

image.png

The responsibility chain + strategy pattern familiar in Java closely resembles this decision loop:

while (goal not complete) {
    think nextStep;
    select tool;
    execute tool.call(params);
    observe toolResult;
    update context;
}

9.2 Three-tier progressive paradigm: don't conflate them

Tier Paradigm What problem it solves Termination mechanism Risk
Inner · reasoning ReAct (Thought → Action → Observation) how a single agent “thinks and moves” model self-judgment easily infinite loops, easily premature termination
Middle · scheduling Loop engineering (graph state machine / multi-agent loops) how a multi-step task “runs to completion and converges” structured rules + state judgment no fault tolerance, no security control
Outer · control Harness engineering (fences, brakes, ops foundation) how production “stays controllable and shippable” global policy —

Formula: production multi-agent = Harness engineering (outer control) + Loop engineering (middle scheduling) + ReAct (inner reasoning)

9.3 A ReAct loop with “brakes and fallbacks”

A demo has only a while(true); in production it dies at the first wall. A loop fit for production must have four layers of protection:

/**
 * Four layers of protection: ① hard step limit ② tool exceptions converted to text and fed back ③ budget (time/token) circuit breaker ④ fallback convergence copy
 */
public Answer run(String userInput, int maxSteps, long budgetMillis) {
    List<ChatMessage> messages = new ArrayList<>();
    messages.add(SystemMessage.from(REACT_PROMPT + renderToolSchemas()));
    messages.add(UserMessage.from(userInput));

    long deadline = System.currentTimeMillis() + budgetMillis;

    for (int step = 0; step < maxSteps; step++) {                    // ① step limit
        if (System.currentTimeMillis() > deadline) {                 // ③ budget circuit breaker
            return new Answer("Task timed out; stopped and returning partial results gathered so far.");
        }

        AiMessage ai = chatModel.generate(messages).content();
        if (!ai.hasToolExecutionRequests()) {
            return new Answer(ai.text());                            // convergence: final answer
        }

        messages.add(ai);
        for (ToolExecutionRequest req : ai.toolExecutionRequests()) {
            String result;
            try {
                result = toolExecutor.execute(req, UserContextHolder.tenantId());
                result = truncate(result, 2000);                     // truncate: stop tool results from blowing the context
            } catch (Exception e) {
                // ② don't let the exception blow through: convert to text, model can take another path
                result = "TOOL_ERROR: " + e.getMessage() + ", switch approaches or tell the user to retry later.";
            }
            messages.add(ToolExecutionResultMessage.from(req, result));
        }
        // over budget, compact history: keep recent N rounds + early summary
        messages = contextCompactor.compact(messages, TOKEN_BUDGET);
    }
    return new Answer("Step limit exceeded; interrupted. Please break the question into smaller pieces.");       // ④ fallback
}

💡 Technical highlight: beyond MAX_STEPS, also do tool-call deduplication — same tool with same parameters short-circuited on consecutive calls; this is the specific remedy for “repeatedly calling the same tool, burning money without converging.” ToolResponse must be truncated; a 100,000-row SQL result stuffed into context is a direct token explosion.

A common context-compaction implementation: record per-round token counts; when over limit, summarize old messages and keep the last few rounds verbatim:

public List<Message> manageContext(List<Message> history) {
    int tokenCount = countTokens(history);
    if (tokenCount < MAX_TOKENS * 0.8) return history;

    // take the first half of old messages and generate a summary
    List<Message> oldMsgs = history.subList(0, history.size() / 2);
    String summary = llm.call("Summarize the following conversation in one sentence:", oldMsgs);

    // new context = summary + the most recent rounds verbatim
    List<Message> newContext = new ArrayList<>();
    newContext.add(new SystemMessage("history summary: " + summary));
    newContext.addAll(history.subList(history.size() / 2, history.size()));
    return newContext;
}

9.4 Framework level: minimal AiServices assembly

Writing the loop yourself is for understanding principles; real projects use the framework, which implements the ReAct loop for you:

@Configuration
public class AgentConfig {

    @Bean
    public CustomerServiceAgent customerServiceAgent(
            ChatLanguageModel chatModel,
            OrderQuerySkill orderSkill,
            LogisticsQuerySkill logisticsSkill) {

        return AiServices.builder(CustomerServiceAgent.class)
                .chatLanguageModel(chatModel)
                .tools(orderSkill, logisticsSkill)                       // inject business skills
                .chatMemory(MessageWindowChatMemory.withMaxMessages(20)) // conversation memory
                .maxIterations(5)                                        // max decision steps, prevent infinite loops
                .timeout(Duration.ofSeconds(30))                         // global timeout
                .build();
    }
    // usage: agent.chat("when will my order 1001 arrive?")
}

Memory splits into two layers: short-term memory uses a sliding window (MessageWindowChatMemory); long-term memory can be persisted to a vector DB and retrieved per user profile.

9.5 Agent vs Workflow: don't use Agent just to use Agent

Workflow has a hardcoded path; Agent's path is decided by the model. Processes that can be solved deterministically should use Workflow — stable and cheap; only when the task path is truly unknowable in advance is Agent worth it. This is a cost-and-stability question, not a technical one.

To evaluate whether an Agent is good, look at these metrics: task completion rate, average steps, tool-call accuracy, end-to-end latency, cost/token, plus trace replay for failure diagnosis.

All concepts are done; how do they collaborate? Watch one complete request.

10. End-to-end walkthrough: one customer service request's full chain

Scenario: e-commerce smart customer service, user asks — “Why hasn't my order SO202610150001 shipped yet?”

image.png

Every concept appears in this chain: Skills decide the flow → LLM does intent recognition → Tool decides the action → MCP connects legacy systems → RAG supplies business knowledge → vectors and cosine similarity handle recall → finally the engineering governance layer guarantees stability, safety, and cost. It's not stuffing all knowledge into the LLM; the Agent decides what to query, what to call, how to answer.

The same skeleton holds in other scenarios:

Knowledge Q&A + business queries + continued executable actions, all three present — that's an enterprise-grade AI application.

11. Engineering governance layer: Java's home turf

The next thing interviewers often want to hear: “If 1000 users ask simultaneously, how does your Spring Boot backend handle it?” Three protections, none optional:

/**
 * ① streaming output (SSE + WebFlux): the LLM is slow, never block synchronously, stream as generated to press down TTFT;
 * ② multi-tenant rate limiting: Redis token bucket per tenant, preventing one tenant from eating the quota;
 * ③ timeout degradation: model timeout must not hang the page, degrade to “knowledge base direct output” to preserve availability.
 */
@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> stream(@RequestParam String q,
                           @RequestHeader("X-Tenant-Id") String tenantId) {

    return redisRateLimiter.acquire(tenantId)                 // ② Redis + Lua token bucket
            .filter(RateLimitResult::allowed)
            .switchIfEmpty(Flux.error(new BizException("TOO_MANY_REQUESTS")))
            .flatMapMany(ok -> chatClient.prompt()
                    .user(q)
                    .stream()                                  // ① streaming
                    .content()
                    .timeout(Duration.ofSeconds(30))           // ③ overall timeout
                    .onErrorResume(e ->                        // ③ degradation fallback
                            Flux.just("Model service busy, returning knowledge base raw retrieval results.")));
}

These numbers should be quotable — they show engineering intuition:

Several other bread-and-butter items, none skippable:

12. Pitfall encyclopedia: technical difficulties vs solutions comparison table

This table merges all front-line pitfalls; save it as a checklist:

# Difficulty Typical symptom Solution Java-side handle
1 Hallucination fabricating order IDs/policies with confidence Prompt hard constraints + forced citation + low temperature + threshold refusal + human fallback; RAG only lowers probability, can't eliminate it system prompt + Advisor + citation backfill
2 Retrieval recall inaccuracy vector similarity ≠ semantic relevance; numbers/terms/names are dense retrieval's weak spot hybrid retrieval (Dense + BM25 → RRF) → Cross-Encoder rerank → top-5~8; Multi-Query rewriting ES one query eating both BM25 + vector; tune similarityThreshold
3 Chunking semantic severing fixed-length hard cut severs arguments structure-aware chunking + 10%~20% overlap + parent-child blocks (child retrieves, parent fed to model) + metadata enhancement pluggable chunker, strategy changes via offline eval
4 Context explosion / cost runaway multi-step Agent token consumption tens of times a normal conversation sliding window + history summary + tool result truncation + Skills on-demand + semantic cache + model tiered routing + stable prefixes to raise KV cache hits Redis semantic cache + gateway token metering + cost dashboard
5 Agent infinite loop / runaway repeatedly calling same tool, going off track, burning money forever hard step limit + time/token dual budget circuit breaker + call deduplication + graph state machine replacing while(true) maxIterations + Spring StateMachine
6 Tool mis-call / parameter errors querying order but calling refund API; parameters filled wrong improve @Tool descriptions + few-shot examples + JSON Schema strict validation + error feedback retry loop + runtime tool set convergence ToolProvider dynamic decision + Bean Validation
7 Tool security / privilege escalation prompt injection, checking others' orders permission filtering pushed to retrieval + tool whitelist + least privilege + audit logs + human-in-the-loop (HITL) for high-risk ops Spring Security tool-level permissions + gateway-transmitted user context
8 MCP productionization HTTP endpoint default no auth; bare exposure is streaking OAuth2/OIDC in front + gateway rate limiting + tool-level permissions + call audit; prefer Streamable HTTP / STATELESS Spring Security + MCP Server + W3C Trace Context
9 High concurrency and latency synchronous blocking fills the thread pool; users wait 8s for first character WebFlux + SSE streaming + virtual threads + parallel recall + multi-tenant token bucket + timeout degrade to knowledge base direct output Redis + Lua token bucket, timeout + onErrorResume
10 Multi-tenant data leakage different customers' documents mixed in one vector DB metadata tenantId pre-filter at retrieval + per-tenant Partition/Collection isolation vector DB filter expressions + request-level tenant context
11 Vector data inconsistency business DB updated, retrieval still old answers Canal/CDC/Debezium subscribing to binlog incremental sync + version comparison + scheduled full reconciliation + forced refresh API event-driven + transactional update
12 Write idempotency retries cause duplicate refunds / duplicate tickets idempotencyKey + business outbox + unique index + second confirmation AOP pre-check + idempotency service
13 Effectiveness unmeasurable “feels okay,” can't regression after go-live golden QA set, quantify recall@k / faithfulness / hallucination rate; Prompt and chunking changes pass regression gate RAGAS / TruLens in CI
14 Fragile tech stack models, protocols, frameworks iterate monthly (MCP spec overhauled within a year, SSE deprecated) lock change inside an adapter layer: unified ChatModel abstraction, tools via MCP standard protocol, Prompt versioning interface abstraction + config-driven + contract tests
15 Multi-turn context loss “what's its standard?” doesn't know what “it” refers to rewrite question based on conversation history, fill references before retrieval Query Rewrite component
16 Data security sensitive data enters model gateway masking + permission filtering + private deployment gateway filters + data classification

13. Quick Q&A: the interviewer's follow-up barrage

# Question One-sentence answer
1 Why not Euclidean distance for cosine similarity? Text semantics cares more about direction than length; magnitude carries text-length noise; after normalization, ranking is equivalent
2 Why use inner product after normalization? saves two square roots and one division per query; retrieval is a hot path, benefit shows directly in P99
3 Vector DB vs ES difference? vector DB specializes in ANN and high-dim retrieval; ES excels at “one system doing text and vectors,” hybrid retrieval easiest. Choice is ops cost and scale
4 With 1M context, still need RAG? Long context is a budget, not a replacement. More material = more noise, higher cost and latency; RAG solves “filter first, then feed”
5 RAG vs fine-tuning? frequent knowledge updates, citation traceability → RAG; fixed style, professional format, compressed prompt → fine-tuning. Production commonly combines: small-model fine-tuning + RAG retrieval + Agent tools
6 Agent vs Workflow? Workflow path hardcoded; Agent path decided by model. Use Workflow for deterministic processes; don't use Agent just to use Agent
7 MCP vs Function Calling? FC is model-side “decide which function to call,” MCP is engineering-side “how functions connect standardly.” MCP gives tools implement-once, reuse-many
8 Skills vs Prompt? Prompt is this time what to do; Skills are this class of tasks by what process, progressive disclosure makes “load many without occupying context” possible
9 Why need Rerank? vector recall is two-tower independent encoding, can't interact deeply; Rerank cross-encoding accurate but slow, so only rerank Top-50
10 How many vector dimensions? common 768/1024/1536/3072; higher dims finer expression but linear cost, enterprise commonly 1024 + PQ quantization
11 Model down, what then? three-layer fallback: multi-key/multi-region retry → switch backup model → knowledge base raw output. Availability from architecture, not the model
12 How to prove RAG works? golden QA set, quantify recall@k / faithfulness / hallucination rate, Prompt and chunking changes must pass regression gate
13 Java for AI stronger than Python where? model layer Python ecosystem better, but engineering layer Java stronger: virtual threads for high concurrency, strong typing fits structured output, mature microservice/circuit-breaker/observability stack, and 90% of enterprise business systems are already Java — MCP Server hooks directly onto existing services, no rewrite

14. Closing: a quick reference table + three pieces of advice

Concept quick reference, save it:

Concept One-sentence essence Key parameters / focus Java landing
LLM probability generation model, not knowledge base temperature, maxTokens, context budget ChatModel / ChatClient
Token billing and length unit Chinese 1 char ≈ 11.5 tokens gateway-side metering and quota
Embedding turns semantics into high-dim coordinates dimensions, whether L2-normalized, model consistency EmbeddingModel.embed()
Cosine similarity direction-only similarity [-1,1]; normalized == inner product hand-written fallback + hand to vector DB
Vector DB storage and ANN retrieval of high-dim vectors recall@K, HNSW efSearch, IVF nprobe Milvus / pgvector / ES 8.x
RAG external knowledge management + retrieval + generation topK, threshold, chunking strategy, Rerank QuestionAnswerAdvisor
Tool Calling model names, program executes description quality, parameter Schema, permissions @Tool / ToolCallback
Agent decides what to do next itself step limit, budget circuit breaker, termination condition AiServices + state machine
MCP unified socket for AI tools Tools/Resources/Prompts; Streamable HTTP spring-ai-starter-mcp-server-*
Skills on-demand-loaded behavior manual progressive disclosure, SKILL.md < 500 lines, reference one level deep directory + on-demand injection

Three pieces of advice, bought with real money:

  1. Govern the LLM as an “unreliable third-party service” — timeout, retry, circuit breaker, degradation, observability, not one of the old tricks can be skipped; only this time you're handling probabilistic output;
  2. Use RAG to anchor answers in facts and eval sets to turn effectiveness into numbers — do these two and a project moves from Demo to production;
  3. Skills, MCP, Agent are essentially object-oriented thinking reused in the AI era: encapsulation, abstraction, standardized interfaces. Understand this and no new vocabulary will panic you.

One sentence to remember: the essence of AI applications is putting “uncertain model capability” inside a “deterministic engineering system.” And that happens to be what Java engineers are best at. Few build engines; building the whole car, repairing the highway, and building gas stations is most people's battlefield.

Let's get concrete: in RAG or Agent rollout, what's the most painful pitfall you've hit? Chunking severed semantics, retrieval misfired, or the model confidently fabricating? Chat about your scenario and parameters in the comments; I'll read and reply to each one.

If this big-picture map connected the concepts for you, hit like, share, and follow — practical Java AI engineering content keeps coming, like hybrid retrieval tuning field notes and Agent state machine design. Don't miss it.