Why Chinese Text Diffs Flag the Whole Paragraph: A Design for Algorithms and Protocols
For any editor or collaboration tool serving CJK users, this design eliminates the misleading all-red highlight and keeps diff results reproducible and attribution honest. Western developers building rich-text editors with ProseMirror or Tiptap can directly apply the same algorithm-protocol split to their own diff features.
Most diff tools were built for space-delimited languages, so a one-character edit in Chinese can surface as a full paragraph delete-and-insert. The fix isn't just smarter segmentation; it's a layered approach: tokenize words and then characters for CJK, compare at block level before drilling into inline content, and treat a similar delete-plus-insert pair as a rewrite rather than two operations.
Beyond the algorithm, the design draws a hard line between computation and protocol. Diff marks live only in editor plugin state, never in the document, so they don't sync, enter undo history, or get saved. Positions are recomputed from scratch on every edit instead of shifted, because a small change can turn a "whole paragraph rewritten" conclusion into "barely changed." Attribution follows the same principle: it records who added new content (an increment), not a census of who wrote every character, and it refuses to guess when data is missing.
The article's central claim is that diff is not just an algorithmic problem but a protocol problem; the hardest decisions are about identity and boundaries, not faster loops.
Keeping diff marks out of the document is a deliberate anti-pattern against naive state management; it prevents collaboration and undo pollution at the cost of recomputing on every keystroke.
Attribution as 'increment' rather than 'census' is a sharp semantic choice that avoids crediting old paragraph creators for new edits, but it also means deletions stay anonymous by design.
The 'recompute from scratch' stance sacrifices performance for correctness and reproducibility, which is notable in an era when many tools optimize for incremental updates.
The article's advice to always prefer 'delete' on ties ensures deterministic output, but it may bias diff markers toward showing more deletions than insertions in ambiguous cases.
Grapheme-level segmentation for combining characters is one open requirement. Stateless recomputation wins over shifting old markers because it can express that the conclusion changed, and a fixed tie-breaking rule that always prefers deletion keeps diff marks stable between refreshes. The remaining concern is performance: full recompute on very large documents might trigger budget downgrades so often that users see only coarse whole-paragraph delete-and-add results.
Combining characters also need to be split at the grapheme level.
The part 'difference is an observation, not content' is what I most agree with. Marks only go into the plugin state, recompute from scratch each time, preferring to compute an extra time rather than shifting old marks—this trade-off is very clean. Shifting looks cheaper, but actually 'the conclusion changed' cannot be expressed. Another point I want to add: 'when tied, always prefer deletion' looks like a detail, but without it, the same pair of documents would produce a different set of markers after each refresh, and the diff tool's credibility would be gone immediately. A question: for tiptap-myers-content-diff, what magnitude of time does a full recompute take on a very large document? If every keystroke triggers a full recompute, would the frequency of budget downgrades be high enough that users often see the coarse result of 'whole paragraph delete + whole paragraph add'?