Keyword Beats Vector on Chinese Tech Notes: 53% vs 40% in a 24-Config Chunking and Recall Test
Same knowledge base, 4 chunking methods × 3 recall methods: pure keyword beats pure vector by 13 points
What this run covered
31 Chinese technical notes as the corpus, 30 questions with a single unique answer as the yardstick, four chunking methods times three recall methods, a total of 24 configurations. All offline, no API key needed.
The results contained three things that differed from my expectations before the run.
| Observation | Number |
|---|---|
| Pure keyword recall is more accurate than pure vector | 53% vs 40% |
| Sentence-level chunking drops table completeness to 6%, while hit rate only drops 6 points | 67% vs 73% |
| Pure keyword scores zero on cross-block questions | 0% |
I also hit two pitfalls, both of the "looks like it works, actually doesn't" kind.
First, a set of numbers
The same 563 chunks, the same 30 questions, three recall methods, and a fixed context budget of 1,200 characters.
| Recall method | Overall | Literal | Paraphrase | Structure | Cross-block | Confusable |
|---|---|---|---|---|---|---|
| Pure keyword | 53% | 75% | 38% | 83% | 0% | 50% |
| Hybrid (0.6 / 0.4) | 73% | 75% | 75% | 100% | 25% | 75% |
| Pure vector | 40% | 50% | 50% | 33% | 25% | 25% |
For literal and structure questions, keyword alone scores 75% and 83%, while pure vector only gets 50% and 33%. For paraphrased questions it's reversed: keyword 38%, vector 50%. For cross-block questions, keyword gets none, vector and hybrid each get 25%, and neither solves it.
So the default assumption that "for RAG you need a vector store" doesn't hold up on this corpus. The division of labor in hybrid recall isn't particularly clever either: keyword handles proper nouns and code blocks, vector handles synonymous rewrites. Both are weak on cross-block, and I'll address that separately later.
First, make "correct answer" judgeable
The percentages above are computable only because each question has exactly one answer and the script can judge right or wrong.
The criterion uses "source file + unique phrase", not block IDs. Block IDs change with the chunking method; if we used them as the criterion, changing the splitter would invalidate all 30 questions. But this article is exactly about changing the splitter.
{
"id": "q-07",
"type": "structure",
"question": "What was the auditDup printed by the concurrent probe in pit-17?",
"expected": {
"source": "…/pit-17-idempotency-check-then-write-all-leaks-under-concurrency.md",
"phrases": ["auditDup=0"]
}
}
The definition of a hit is simple. Within the top k retrieved chunks, if any chunk's source is in the allowed set and its body contains at least one of the answer phrases, it's a hit.
The 30 questions fall into five categories, each testing a specific concern.
| Type | Count | What it tries to prove |
|---|---|---|
| Literal | 8 | Baseline. If these don't hit, the index is broken |
| Paraphrase | 8 | Semantic ability. The question doesn't copy the original words |
| Structure | 6 | Main conclusion. The answer is only in a code block or table |
| Cross-block | 4 | The answer needs two or more chunks pieced together |
| Confusable | 4 | Words in the question appear in multiple places; can it pick the right file |
There is one rule that must be written into the test cases. The answer phrase must appear exactly once in its own file. If it appears zero times, the criterion always fails; if it appears twice, retrieving another block also counts as a hit, inflating the hit rate without the program reporting an error. I ran a reverse check on this: replacing the phrase with a string that appears twice immediately turned the test red and pointed out which one it was.
Another complication is duplicate copies. Published articles have copied the original notes, so the same conclusion appears scattered across two to four files. Recognizing only one source would artificially lower the hit rate, so I added "equivalent sources" to the criterion, allowing a group of files to count. Confusable questions deliberately exclude this, because that category tests whether you can pick the right file.
Spread out the 24 configurations
Three things stay fixed. Local embedding uses bge-small-zh, so the same chunk always gets the same vector; the context budget is 1,200 characters; the 30 questions never change.
The budget is the tightest constraint here. The larger the chunk, the more characters fit when you take k chunks, and the higher the hit rate naturally rises. Without a budget limit, the optimal solution is always "put the whole article in one chunk", which is arithmetic, not a conclusion. Pin the budget to 1,200 characters, and the four chunking strategies stand on the same scale.
| Chunking | Size/Overlap | Chunks | Recall | Hit rate | literal | paraphrase | structure | crossblock | confusable | Avg context | Table completeness | Code completeness |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| By character | 300/30 | 431 | Vector | 33% | 38% | 25% | 50% | 50% | 0% | 1180 chars | 55% | 55% |
| By character | 300/30 | 431 | Hybrid | 60% | 75% | 50% | 83% | 50% | 25% | 1181 chars | — | — |
| By character | 300/30 | 431 | Parent chunk | 53% | 75% | 50% | 50% | 50% | 25% | 984 chars | — | — |
| By character | 800/80 | 172 | Vector | 30% | 25% | 38% | 17% | 75% | 0% | 782 chars | 75% | 80% |
| By character | 800/80 | 172 | Hybrid | 47% | 25% | 50% | 67% | 100% | 0% | 792 chars | — | — |
| By character | 800/80 | 172 | Parent chunk | 47% | 63% | 38% | 50% | 50% | 25% | 631 chars | — | — |
| By paragraph | 300/30 | 556 | Vector | 43% | 50% | 50% | 50% | 25% | 25% | 1077 chars | 68% | 82% |
| By paragraph | 300/30 | 556 | Hybrid | 70% | 75% | 75% | 100% | 25% | 50% | 1057 chars | — | — |
| By paragraph | 300/30 | 556 | Parent chunk | 43% | 63% | 25% | 50% | 50% | 25% | 1899 chars | — | — |
| By paragraph | 800/80 | 196 | Vector | 30% | 50% | 25% | 17% | 50% | 0% | 839 chars | 94% | 95% |
| By paragraph | 800/80 | 196 | Hybrid | 67% | 63% | 63% | 83% | 100% | 25% | 752 chars | — | — |
| By paragraph | 800/80 | 196 | Parent chunk | 50% | 63% | 50% | 50% | 50% | 25% | 705 chars | — | — |
| By sentence | 300/30 | 453 | Vector | 37% | 38% | 38% | 33% | 50% | 25% | 1124 chars | 6% | 18% |
| By sentence | 300/30 | 453 | Hybrid | 67% | 75% | 75% | 83% | 50% | 25% | 1139 chars | — | — |
| By sentence | 300/30 | 453 | Parent chunk | 50% | 75% | 38% | 50% | 50% | 25% | 1239 chars | — | — |
| By sentence | 800/80 | 191 | Vector | 33% | 25% | 38% | 17% | 75% | 25% | 810 chars | 45% | 69% |
| By sentence | 800/80 | 191 | Hybrid | 57% | 38% | 63% | 83% | 100% | 0% | 795 chars | — | — |
| By sentence | 800/80 | 191 | Parent chunk | 50% | 63% | 50% | 50% | 50% | 25% | 792 chars | — | — |
| Recursive | 300/30 | 563 | Vector | 40% | 50% | 50% | 33% | 25% | 25% | 1091 chars | 70% | 80% |
| Recursive | 300/30 | 563 | Hybrid | 73% | 75% | 75% | 100% | 25% | 75% | 1068 chars | — | — |
| Recursive | 300/30 | 563 | Parent chunk | 43% | 63% | 25% | 50% | 50% | 25% | 1918 chars | — | — |
| Recursive | 800/80 | 196 | Vector | 30% | 50% | 25% | 17% | 50% | 0% | 851 chars | 94% | 95% |
| Recursive | 800/80 | 196 | Hybrid | 67% | 63% | 63% | 83% | 100% | 25% | 752 chars | — | — |
| Recursive | 800/80 | 196 | Parent chunk | 50% | 63% | 50% | 50% | 50% | 25% | 705 chars | — | — |
Four things come out of this table.
The recall method has more leverage than the chunking method. Under the same chunking configuration, switching from pure vector to hybrid adds 17 to 37 points (the recursive 300/30 group goes from 40% straight to 73%). But chunking can't be ignored either; under the same recall method, switching chunking produces a max difference of 20 points (800/80 hybrid: by character 47%, by paragraph and recursive both 67%).
Structural completeness and hit rate are decoupled. Sentence-level chunking drops table completeness to 6% and code to 18%, effectively cutting every line apart. Yet under hybrid recall, the hit rate is still 67%, only 6 points below the best. The reason isn't hard to guess: most answer fragments land in ordinary text lines, and destroying structure doesn't mean the answer can't be found. But a model handed a mangled table still can't read it; looking only at hit rate, this loss gets missed.
Parent chunks are not a silver bullet. The recursive and paragraph 300/30 parent chunks have average contexts of 1,918 and 1,899 characters respectively, both over budget (the first chunk is force-included), and their hit rates are only 43%. When sub-chunks get larger, parent chunks shrink to 705 characters, and the hit rate stays below hybrid. Its value lies in context completeness, not hit rate.
Chunk size is a tradeoff. 300 beats 800 on hit rate (hybrid 73% vs 67%), while 800 has much higher structural completeness (table 94% vs 70%). You pay a price at either end.
Why keyword is stronger on this corpus
This conclusion depends on the nature of the corpus; don't transplant it to other scenarios directly.
This corpus is Chinese technical notes, with an extremely high density of proper nouns like 幂等键, defaultToolCallbacks, on-http-codes. The vector model squeezes them into 512 dimensions, so adjacent concepts tend to blur together; idempotency, grabbing a slot, and rate limiting all count as "concurrency and guardrails", semantically very close. Bigram matching almost never misses on proper nouns, at the cost of being completely incapable on paraphrases, which score only 38% in that column.
The cross-block column at 0% for keyword deserves its own note. It can only find the block whose words match; it can't piece together an answer across segments. Hybrid only gets 25% as well, so cross-block is the weak spot for every method. To actually solve it, you'd need parent chunk backfill or multi-round retrieval, but parent chunk backfill isn't cost-effective within the 1,200-character budget, as the earlier average-context numbers above have shown.
Three parameter values I guessed, later swept
The hybrid weight 0.6/0.4, budget 1,200 characters, and similarity threshold 0 were all initially my guesses. Guessed values can't be taken as conclusions, so I swept each one later to see whether it was a plateau or a lucky point.
| Vector / keyword | Hit rate |
|---|---|
| 0.0 / 1.0 | 53% |
| 0.2 / 0.8 | 67% |
| 0.3 / 0.7 | 67% |
| 0.4 / 0.6 | 73% |
| 0.5 / 0.5 | 73% |
| 0.6 / 0.4 | 73% |
| 0.7 / 0.3 | 63% |
| 0.8 / 0.2 | 60% |
| 1.0 / 0.0 | 40% |
The three levels from 0.4 to 0.6 give identical results, so any one of them is the same. Going down to 0.3 drops to 67%, going up to 0.7 drops to 63%. It's a wide plateau, not a lucky point.
| Budget (chars) | Hit rate | Actual average chars used |
|---|---|---|
| 300 | 30% | 242 |
| 600 | 50% | 517 |
| 900 | 57% | 770 |
| 1200 | 73% | 1067 |
| 1800 | 77% | 1708 |
| 2400 | 80% | 2281 |
| 4000 | 83% | 3892 |
From 900 to 1,200 adds 16 points, from 1,200 to 1,800 only adds 4 points, after 1,800 every doubling trades for 3 points. The 1,200 mark is the "more isn't worth it" point.
| minScore | Hit rate | Average chunks retrieved |
|---|---|---|
| -1.0 | 73% | 4.6 |
| 0.0 | 73% | 4.6 |
| 0.2 | 73% | 4.6 |
| 0.3 | 73% | 4.6 |
| 0.4 | 73% | 4.6 |
| 0.5 | 60% | 4.4 |
| 0.6 | 30% | 3.2 |
Between 0 and 0.4 the results are identical. Chunks that make the budget already have cosines above 0.4, so the threshold never filters anything. Pushing it to 0.5 starts false-killing, dropping the hit rate to 60%, and at 0.6 it's only 30%. So on this corpus setting the threshold to 0 is correct, and setting it anyway is pointless.
Also checked Python's defaults
Many people come over from Python. I unpacked the source of langchain_text_splitters 0.3.11 to look at the defaults, and there are quite a few mismatches with the Java side.
| LangChain (Python) | LangChain4j (Java) | |
|---|---|---|
| Default chunk size | chunk_size=4000, chunk_overlap=200 |
No default; must pass it |
| Splitting basis | Character (length_function=len) |
Character, can switch to TokenCountEstimator |
| Default separators | ["\n\n", "\n", " ", ""] |
Paragraph, line, sentence, word, character; chained fallback |
| Language presets | from_language(Language.JAVA) splits on \nclass , \npublic , \nif etc. |
None; you have to write a DocumentByRegexSplitter yourself |
I confirmed the Java chain using javap -c on the jar. DocumentSplitters.recursive sequentially constructs four splitters for paragraph, line, sentence, and word, where the word level degrades to character; there's no such thing as a "separator list".
Three places you can't just copy over.
- The 4,000-character default is too long for Chinese. In the matrix above, 800 is already the "structure-completeness-first" level, and 300 has a higher hit rate. If you directly apply 4,000, one chunk crams in several sections, and retrieval precision is guaranteed to drop.
- The
" "level is almost useless for Chinese. Chinese doesn't tokenize by spaces, that level basically never triggers; what actually applies are newlines and the character level. - Java has no language presets. To split by class or by method, you have to write your own regex. If the regex is wrong, it doesn't error; the chunks just "look normal".
Two pitfalls
Pitfall one: Intel Macs can't install local Chinese embeddings. The error message looks like this.
ai.djl.engine.EngineException: Failed to load Huggingface native library.
Caused by: ai.djl.engine.EngineException: Unexpected flavor: cpu
"Unexpected flavor: cpu" led me to the ONNX runtime flavor configuration; I spent a long time checking environment variables and found nothing. Then I unpacked the jar and looked at native/lib/, and found the real cause. DJL's tokenizers stopped publishing the osx-x86_64 native library starting from 0.30.0, while LangChain4j defaults to pulling 0.36.0, which has no matching .dylib for an Intel Mac. Unpacking version by version confirmed that 0.29.0 is the last version that still ships this library. Pinning both ai.djl:api and ai.djl.huggingface:tokenizers to 0.29.0 made the test pass. Pinning only one causes a version mismatch.
Pitfall two: matches() in a regex made splitting produce only one chunk. The first version's output looked like this.
Candidate chunks: all 31, worth asking 31, involving 31 documents
Every document produced exactly one chunk. Pattern.matches() requires the entire line to match completely, but my regex had a line-start anchor. Changing to find() gave 237 chunks. The danger here is that the output looks completely normal, with chunk counts equal to document counts, no error, no warning.