跪拜 Guibai
← All articles
Frontend · JavaScript · TypeScript

Why Chinese Text Diffs Flag the Whole Paragraph: A Design for Algorithms and Protocols

By 泯泷 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

For any editor or collaboration tool serving CJK users, this design eliminates the misleading all-red highlight and keeps diff results reproducible and attribution honest. Western developers building rich-text editors with ProseMirror or Tiptap can directly apply the same algorithm-protocol split to their own diff features.

Summary

Most diff tools were built for space-delimited languages, so a one-character edit in Chinese can surface as a full paragraph delete-and-insert. The fix isn't just smarter segmentation; it's a layered approach: tokenize words and then characters for CJK, compare at block level before drilling into inline content, and treat a similar delete-plus-insert pair as a rewrite rather than two operations.

Beyond the algorithm, the design draws a hard line between computation and protocol. Diff marks live only in editor plugin state, never in the document, so they don't sync, enter undo history, or get saved. Positions are recomputed from scratch on every edit instead of shifted, because a small change can turn a "whole paragraph rewritten" conclusion into "barely changed." Attribution follows the same principle: it records who added new content (an increment), not a census of who wrote every character, and it refuses to guess when data is missing.

Takeaways
— English text splits naturally by spaces, but Chinese, Japanese, and Korean have no spaces, so naive tokenizers treat a whole paragraph as a single token.
— The tokenizer first segments by language rules and then further splits CJK words into individual characters; spaces are kept as separate blocks to preserve position alignment.
— The core diff algorithm is the classic shortest edit script, with three protocol-level enhancements: trim common head and tail, cap the edit distance budget to avoid UI freezes, and always prefer 'delete' on ties for reproducible results.
— A document is compared hierarchically: first pair blocks (paragraphs, headings, table rows), then drill into block internals only when necessary; a similar delete-plus-insert pair is re-paired as a rewrite.
— Style changes are handled by ignoring styles during pairing, then checking style differences after a successful match, so bolding a word reads as a style change, not a delete-and-insert.
— Deleted content has no position in the new document, so it is serialized back as a read-only 'patch block' rendered with its original formatting (bold, tables, images).
— All diff marks live only in plugin state and are never written into the document, so they don't sync, enter undo history, or get saved.
— Positions are recomputed from scratch on every edit rather than shifting old marks, because a single character change can flip a whole-paragraph rewrite back to a minor tweak.
— Attribution only answers who added new content (an increment), not who wrote each character; if unknown, it returns null rather than guessing.
— Deletions almost never have attribution because most collaboration systems don't record who deleted; the design accepts this rather than fabricating a name.
— The whole-version case (e.g., rollback) uses whole-version attribution, crediting the person who triggered the rollback, not the original author.
— The algorithm handles speed and accuracy; the protocol defines what the result means, where boundaries lie, and how it should be interpreted.
Conclusions

The article's central claim is that diff is not just an algorithmic problem but a protocol problem; the hardest decisions are about identity and boundaries, not faster loops.

Keeping diff marks out of the document is a deliberate anti-pattern against naive state management; it prevents collaboration and undo pollution at the cost of recomputing on every keystroke.

Attribution as 'increment' rather than 'census' is a sharp semantic choice that avoids crediting old paragraph creators for new edits, but it also means deletions stay anonymous by design.

The 'recompute from scratch' stance sacrifices performance for correctness and reproducibility, which is notable in an era when many tools optimize for incremental updates.

The article's advice to always prefer 'delete' on ties ensures deterministic output, but it may bias diff markers toward showing more deletions than insertions in ambiguous cases.

Concepts & terms
Shortest Edit Script
The minimum sequence of insert and delete operations that transforms one sequence into another; identical elements can be kept as free diagonal moves in an edit grid, making it the classic core of diff algorithms.
Tokenization for CJK
Languages like Chinese, Japanese, and Korean do not use spaces between words, so a tokenizer must rely on language rules to segment words and then optionally split further into individual characters to make diffs meaningful.
Plugin state vs. document state
A design principle in editor architecture where diff marks live outside the document model (e.g., in ProseMirror plugin state), so they don't propagate to collaborators, undo history, or saved content.
Increment-based attribution
A way to record authorship that tracks which ranges of a new document were added in the current edit, rather than conducting a full census of who wrote every character; it avoids crediting the original creator of a paragraph for a later edit.
From the discussion

Grapheme-level segmentation for combining characters is one open requirement. Stateless recomputation wins over shifting old markers because it can express that the conclusion changed, and a fixed tie-breaking rule that always prefers deletion keeps diff marks stable between refreshes. The remaining concern is performance: full recompute on very large documents might trigger budget downgrades so often that users see only coarse whole-paragraph delete-and-add results.

— Combining characters demand grapheme-cluster segmentation, not just word and character splitting.
— Recomputing marks from scratch is preferable to shifting old markers, since shifting cannot represent a changed conclusion.
— Always preferring deletion on ties is essential; without it the same pair of documents yields a different set of marks after each refresh and the tool loses trust.
— Full recompute cost on huge documents is the open question—if every keystroke recomputes, budget downgrades may surface coarse whole-paragraph diff results frequently.
Featured comments
卷土的土

Combining characters also need to be split at the grapheme level.

组合搭配函数

The part 'difference is an observation, not content' is what I most agree with. Marks only go into the plugin state, recompute from scratch each time, preferring to compute an extra time rather than shifting old marks—this trade-off is very clean. Shifting looks cheaper, but actually 'the conclusion changed' cannot be expressed. Another point I want to add: 'when tied, always prefer deletion' looks like a detail, but without it, the same pair of documents would produce a different set of markers after each refresh, and the diff tool's credibility would be gone immediately. A question: for tiptap-myers-content-diff, what magnitude of time does a full recompute take on a very large document? If every keystroke triggers a full recompute, would the frequency of budget downgrades be high enough that users often see the coarse result of 'whole paragraph delete + whole paragraph add'?

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗