Java · Spring Boot · AI

Keyword Beats Vector on Chinese Tech Notes: 53% vs 40% in a 24-Config Chunking and Recall Test

By Java内核笔记 · Oct 26, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Vector stores are not the default win in retrieval: on a corpus dense with domain-specific proper nouns, a simple keyword matcher can outperform full embeddings by 13 points, and hybrid recall adds another 20. Anyone building RAG on code-heavy or jargon-heavy Chinese material should sweep chunking and recall combinations before committing to a vector index, and the two-pitfall list alone could save hours of debugging on Java deployments.

Summary

A systematic offline test on 31 Chinese technical notes and 30 single-answer questions ran 24 configurations (4 chunking strategies × 3 recall methods) with a fixed 1,200-character context budget. Pure keyword retrieval beat pure vector retrieval overall (53% vs 40%), but a hybrid mix (0.6 keyword / 0.4 vector) topped the table at 73%. The gap was most visible on literal and structure questions, where keywords handled proper nouns and code blocks cleanly; vector only won on paraphrased questions. Cross-block questions exposed the weakest spot, with all methods scoring at or below 25%.

Takeaways

— On the 31-document Chinese technical note dataset, pure keyword recall scored 53%, pure vector 40%, and hybrid (0.6/0.4) 73%.

— Pure keyword matched 75% of literal questions and 83% of structure questions; pure vector managed 50% and 33% respectively.

— On paraphrased questions, keyword scored 38%, vector 50%—the only category where vector won.

— Cross-block questions (answers needing two or more chunks) were unsolved across the board: pure keyword 0%, hybrid 25%, pure vector 25%.

— Sentence-level chunking cut table completeness to 6% and code completeness to 18%, yet hybrid retrieval still reached 67% overall, only 6 points below the best configuration.

— The hybrid weight sweep showed a flat plateau from 0.4/0.6 to 0.6/0.4 (all 73%); it drops to 67% at 0.3/0.7 and 63% at 0.7/0.3.

— Increasing the context budget from 900 to 1,200 characters added 16 points (57% to 73%), but from 1,200 to 1,800 only added 4 points; after that, doubling the budget buys about 3 points.

— The similarity threshold had no effect between 0 and 0.4 (73% constant) because retrieved blocks already had cosines above 0.4; it started hurting at 0.5.

— Python's langchain_text_splitters default chunk_size=4000 and chunk_overlap=200, but LangChain4j has no default chunk size and no language presets; the recursive splitter in Java uses a chain of paragraph/line/sentence/word fallback, not a list of separators.

— Two pitfalls: Intel Macs fail to load local Chinese embeddings because DJL tokenizers dropped osx-x86_64 native libraries after 0.30.0 (pin to 0.29.0), and using Pattern.matches() instead of find() in a regular expression silently produced one chunk per document.

— The answer-judging criteria used 'source file + unique phrase' rather than block IDs, because block IDs change with the splitter; the phrase must appear exactly once in its source file, and equivalent sources were allowed for duplicate published content, except for confusable questions.

Conclusions

On a corpus dense with technical terms like 幂等键 and defaultToolCallbacks, keyword matching is not a fallback; it is often the most reliable exact-match mechanism, while dense vectors smudge nearby concepts into the same semantic neighborhood.

The finding that sentence-level chunking can still hit 67% recall with a hybrid method, despite destroying table structure, suggests that answer fragments often live in plain text lines, but the destroyed structure still hurts the downstream model's comprehension in ways a recall metric cannot show.

Cross-block questions were the worst scoring category across every method (pure keyword 0%, hybrid 25%), which points to a real limitation of single-pass top-k retrieval when an answer has to be assembled from multiple blocks.

The parameter sweeps show that retrieval performance is not a knife-edge: the hybrid weight plateau between 0.4/0.6 and 0.6/0.4, and the minScore threshold does nothing until it passes 0.4, so naive grid searches on these knobs buy little.

Concepts & terms

Hybrid retrieval

A retrieval method that combines keyword-based (lexical) matching with vector-based semantic search, typically by weighted score fusion. In this test a 0.6/0.4 keyword-to-vector ratio worked best.

Parent chunk recall

A chunking strategy where each block carries a pointer to a larger parent block (e.g., a whole section or document). When a child block is retrieved, the parent block is also returned to supply broader context.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗