跪拜 Guibai
← All articles
Interview · AI Programming · Agent

A RAG Internship Interview That Tested Chunking, Hybrid Search, and Agent Architecture

By 沉默王二 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The interview questions mirror what production AI engineering teams actually ask: not trivia about model names, but tradeoff reasoning about chunking strategies, hybrid search weighting, and cost-aware embedding choices. A candidate who can walk through why BM25 rescues vector recall from semantic drift, or why 2048-dimension embeddings are worth the storage cost, demonstrates the kind of systems thinking that separates demo builders from engineers who ship.

Summary

A student walked into an internship interview and got grilled for 90 minutes on the internals of a production RAG system called Paicongming. The questioning moved from multi-level semantic chunking without overlap and parent-document retrieval, through embedding model selection and cost analysis for 2048-dimension vectors, to the precise mechanics of a hybrid KNN+BM25 recall pipeline where BM25 dominates the final ranking to catch keyword-specific answers that vector search misses.

The interview then shifted to AI engineering practice: how 60% of the project's boilerplate was generated by Codex while core logic required manual tuning, a three-layer verification strategy for AI-generated code, and a crisp breakdown of Tool (atomic function), MCP (standardized tool protocol), and Skill (workflow encapsulation with prompt templates and scripts). The candidate also outlined a ReAct-loop upgrade path to turn the RAG system into an agent with a planner, tool calling, and human-in-the-loop safeguards.

Throughout, the interviewer pressed on tradeoffs: why skip overlap in Chinese text chunking, why let BM25 outweigh KNN, and why DeepSeek over Qwen for the LLM. The answers reveal a team that benchmarked alternatives, calculated storage costs, and learned that RAG effectiveness is 70% data quality, 20% retrieval design, and only 10% model choice.

Takeaways
Multi-level semantic chunking without overlap works better for Chinese text; overlap wastes topK slots by returning duplicate paragraphs.
Maintaining a 1MB parent block alongside small chunks prevents OOM during ingestion and provides full context to the LLM after a chunk hit.
Apache Tika 2.9.1 handles PDF, Word, Excel, PPT, Markdown, and HTML parsing; PDFs get special treatment via PDFBox with page-level slicing and header/footer stripping.
Alibaba Qwen text-embedding-v4 at 2048 dimensions outperformed a competitor by over 10 percentage points on Chinese technical documents.
The extra 8KB per vector at 2048 dimensions was deemed acceptable after calculating storage costs for hundreds of thousands of entries.
Hybrid KNN+BM25 recall uses KNN for coarse candidate retrieval (topK × 30) and BM25 for final re-ranking with a 0.2/1.0 weight split favoring BM25.
BM25 dominates because pure KNN suffers from semantic drift: it retrieves topically similar paragraphs but misses the exact keyword match the user asked for.
BM25 improves on TF-IDF through the k1 parameter, which caps term frequency growth so a word appearing 100 times doesn't score 10× higher than one appearing 10 times.
Production RAG requires roughly 10× the effort of a demo once permission isolation, multi-tenancy, file deduplication, streaming uploads, and incremental updates are added.
AI wrote 60% of the project's code (CRUD, mappers, API wrappers) but the core 40% — chunking strategy, recall tuning, prompt engineering — needed manual iteration.
Verifying AI-generated code uses three layers: AI self-verification via unit tests, multi-agent review on critical paths, and manual testing for stubborn edge cases.
Upgrading RAG to an agent requires iterative retrieval, tool calling (web search, SQL, file I/O), a planner for task decomposition, and production safeguards like timeouts and state persistence.
Tool, MCP, and Skill form a stack: Tool is the atomic function, MCP is the transport protocol, and Skill is a workflow bundling prompts, call sequences, and scripts.
Skill descriptions act as classifiers; if written too broadly, the model triggers them incorrectly, making description precision a critical design detail.
Conclusions

Overlap in chunking is often treated as a default best practice, but this team's data shows it actively harms Chinese retrieval by polluting the topK with near-duplicates — a reminder that NLP defaults from English-centric research don't always transfer.

The decision to let BM25 dominate KNN in hybrid search is counter-trend: most systems treat vector search as primary and keyword search as auxiliary. Their A/B test showed the opposite produces better results for factoid questions where exact term matching matters more than semantic similarity.

The 70/20/10 split (data/retrieval/model) is a useful heuristic that pushes against the instinct to fix RAG problems by swapping models. Most gains come from chunking and retrieval design, which are less glamorous but higher-leverage.

An internship candidate who can discuss HNSW graph structures, BM25 term-frequency saturation, and ES dense_vector tradeoffs signals that Chinese CS programs are producing students with deeper infra knowledge than typical new-grad interview loops test for in Western markets.

The Skill description-as-classifier problem is under-discussed: a poorly scoped description field silently breaks agent behavior by either failing to trigger or triggering spuriously, making it a de facto configuration bug surface that most Skill authors don't test.

Concepts & terms
BM25
A text relevance scoring algorithm (Best Matching 25) that improves on TF-IDF by capping term frequency growth via the k1 parameter and normalizing for document length via the b parameter. Elasticsearch uses it as the default similarity.
HNSW
Hierarchical Navigable Small World, a graph-based algorithm for approximate nearest neighbor search used by Elasticsearch 8.x for dense_vector fields. It builds a multi-layer graph and searches from sparse upper layers down to dense lower layers in O(log n) time.
MCP
Model Context Protocol, a standardized protocol that packages Tools as independent servers callable by any MCP-compatible client (Claude Code, Codex, etc.). It separates tool implementation from tool consumption.
Skill (Claude Code)
A workflow encapsulation in Claude Code consisting of a SKILL.md file with prompt templates and call sequences, a references directory for style guides or samples, and a scripts directory for helper code. Skills are triggered by the model based on their description field.
ReAct
A reasoning-and-acting loop pattern for AI agents where the model alternates between thinking (reasoning about what to do next) and acting (executing a tool call), observing the result, and repeating until a final answer is reached.
Parent Document Retrieval
A RAG pattern where small chunks are used for vector similarity search, but once a chunk is matched, the system retrieves a larger parent block containing that chunk to provide fuller context to the LLM.
From the discussion

The parsing stage of a RAG pipeline draws practical questions: one person wrestling with LaTeX-to-Markdown conversion asks whether Apache Tika's output format suits semantic chunking. On the agent side, model selection emerges as a cost-accuracy tradeoff — Claude leads on accuracy for agent tasks while DeepSeek handles routine automation, and interviewers may probe that selection logic.

LaTeX parsing for RAG is difficult and often requires an intermediate Markdown representation to enable effective semantic chunking.
Apache Tika's output format is an open question — its suitability for downstream semantic splitting is not obvious.
Agent task accuracy is highly model-dependent; Claude delivers the best results but at higher cost, while DeepSeek suffices for standard automation.
Agent-focused interviews may extend beyond architecture into model selection strategy, treating it as a first-class design decision.
Featured comments
MrSkye 1 likes

Lately I've been thinking about adding a RAG project. Just finished the parsing module and ran a benchmark — I wanted to parse LaTeX. Compared a hand-written parsing logic against LaTeXML's parsing in one test. Then realized I'd need to translate it back to Markdown for better semantic chunking, so I started working on that... Parsing really is kind of tricky hahahaha. What format does the output from Apache Tika 2.9.1 that the OP used turn into? Is it suitable for semantic chunking? [thinking]

Easy88AI

Agent performance actually depends heavily on the underlying model choice. In my tests, Claude had the highest accuracy for Agent tasks but was expensive; for routine automation, DeepSeek was fully sufficient. If interviewing for an Agent role, the interviewer might also press you on model selection strategy.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗