Why Chinese Text Diffs Flag the Whole Paragraph: A Design for Algorithms and Protocols
Changing one character makes the whole paragraph red? — Algorithm and protocol design for Chinese content diff
Author: 泯泷
Original: https://mp.weixin.qq.com/s/kbFjNyq6RtCoxKOskcv8Qg (September 17, 2026)
The algorithm is responsible for "calculating accurately," and the protocol is responsible for "what identity it should exist as."
If you've used "version comparison," you've likely seen this: change one character in a sentence, and the tool highlights the entire paragraph, marking it as "delete the old paragraph, add a new paragraph." The screen is full of red and green, and the single character that actually changed is buried.
This isn't a flaw in the tool; it's a matter of its origins. Most diff tools are designed for English — English uses spaces to separate words, and there are natural gaps between words. Chinese, Japanese, and Korean don't use spaces, so a whole paragraph appears as "one word" to them. Change one character, and the entire "word" changes, so the whole paragraph is redone.
The following is my analysis of "content diff" after studying an open-source project (tiptap-myers-content-diff, a diff library for Tiptap / ProseMirror document editors). What's worth discussing is not just "fixing Chinese word segmentation," but that it treats "content diff" as a problem that requires designing both an algorithm and a protocol simultaneously. The algorithm answers "how to calculate quickly and accurately," and the protocol answers "in what form the result exists, who reads it, and where the boundaries are drawn." Below I'll explain both clearly — no need to understand code; each section ends with some TypeScript pseudocode, and skipping it won't affect understanding.
1. First learn to cut: split a paragraph into pieces
The premise of comparison is to cut both sides into small comparable chunks.
An English sentence is like a box of LEGO pieces already divided into cells — spaces are the dividing lines, and you just take them along the lines. Chinese is a single smooth board with no lines; you must make your own cuts. Where you cut determines everything that follows.
Take a minimal example. Change
This is a very long paragraph of text old tail
to
This is a very long paragraph of text new tail
A common old-style tool that splits by spaces will only mark it like this: delete "This is a very long paragraph of text 旧尾" and insert "This is a very long paragraph of text 新尾" — the whole paragraph, together, red and green interwoven.
But a character-level split will mark it like this:
This is a very long paragraph of text
旧新 tail
Only those two characters changed.
Segmentation is usually handled by the system's built-in language capabilities: first get the boundaries of "words," and for languages like Chinese that don't use spaces, further refine words down to characters. This approach requires no additional dependencies.
The same text: on the left treated as a single block (one token), on the right split into many small blocks (many tokens), only two small blocks actually change.
TYPESCRIPT
// Split a piece of text into comparable small blocks
function tokenize(text: string): string[] {
// First get the boundaries of `words`; the basis is the text itself, unrelated to `who the reader is`
const words = segmentByLanguageRules(text)
// Chinese, Japanese, Korean have no spaces, so further refine to `characters`
return words.flatMap(word => (isCJK(word) ? [...word] : [word]))
// Note: spaces must also be kept as separate blocks, otherwise all subsequent positions will be misaligned
}
There's also a protocol-level convention that's easy to overlook: cutting must be independent of "who the reader is." For the same pair of documents, no matter who compares them or on what device, the cut blocks must be exactly the same; otherwise two people looking at the same comparison will count different numbers of "changes." Word boundaries are properties of the text itself, not of the reader.
By the way, spaces are also kept as separate blocks. Because every subsequent "position" is accumulated from these blocks, if you drop a space, all positions afterwards will shift — positions must be reconstructible exactly.
2. Then learn to pair: find the shortest path
After cutting, you have two sequences of blocks. Next, align them: which blocks exist on both sides (unchanged), which only exist in the new version (inserted), and which only exist in the old version (deleted).
This is the core of diff, and it's a classic old problem: the shortest edit script. In plain terms, go from the old text to the new text in the fewest steps — each step either deletes a block or adds a block, and identical blocks on both sides can "slide diagonally" for free. Think of it as a grid map where you go from one corner to another, saving steps as much as possible; identical text is a free diagonal slide on the map.
A shortest path in the edit grid: blue is the "equal" slide, purple is an "insert" step, amber is a "delete" step. The whole path is the most economical way from the old text to the new text.
The algorithm is old, but the real skill lies in three engineering decisions around it, and they are essentially protocol decisions.
First, trim the common head and tail. A long document changed in only one place, and the actual difference is only in a small middle section. First remove the obviously identical beginning and ending, focusing all effort on the truly different parts — this also makes everyday inputs like "change one character" nearly effortless.
Give the algorithm a budget, and if it can't compute, degrade gracefully. If two contents are completely unrelated and the required steps would explode, don't force computation to the point of freezing the page. It's better to degrade to a coarse-grained result like "delete entire paragraph + add entire paragraph." Coarse results are acceptable; a stuck interface is not — "never freeze the user" is a hard constraint.
Make ties deterministic. When multiple equally short optimal paths exist, always prefer "delete" first. This seems nitpicky but is actually the lifeline of diff: if it's not fixed, the same pair of documents, the same person, might get a different markup after refreshing. A non-reproducible comparison is equivalent to no comparison.
TYPESCRIPT
// Shortest edit script: transform left into right with the fewest `insert`/`delete` operations
type Edit<T> = { kind: 'equal' | 'insert' | 'delete'; node: T }
function diffSequence<T>(left: T[], right: T[], keyOf: (x: T) => string): Edit<T>[] {
const { head, middle, tail } = trimCommonEnds(left, right, keyOf) // Trim common head and tail first
const aligned = shortestEdits(middle, {
maxEditDistance: 1024, // Budget: if it can't compute, degrade to `delete all and reinsert` without freezing the UI
})
return [...head, ...aligned, ...tail] // Ties always prefer `delete` to ensure reproducibility
}
An algorithm must not only be "correct" but also "stable" and "flexible."
3. Drill into the structure: compare not only at the character level, but also at the block level
A document is not a single long rope; it has layers: an article has paragraphs, paragraphs have bold, links, images, and also tables, lists, and quotes.
Comparing only at the outermost "character by character" level flattens the overall picture of a structured document. A robust approach is layered drilling: first compare "blocks" (paragraphs, headings, table rows and cells...), and for paired blocks, drill into their internals for word-by-word comparison. Things that can be explained at the block level don't go deeper; only when it's necessary to be precise to the word do you continue downward. It's like first looking at the skeleton of the whole article, then zooming into the paragraph, sentence, or word you actually care about.
On the left is the document's skeleton (blocks); on the right, a paragraph is enlarged and split into words — only drill down to word granularity where needed.
TYPESCRIPT
// Compare `blocks` first; paired blocks drill into their internals; stay at block level when possible
function compareChildren(before: Node, after: Node, out: Change[]): void {
for (const pair of pairRewrites(pairChildren(before, after))) {
if (pair.kind !== 'equal') {
out.push(report(pair)) // insert / delete / rewrite
} else if (pair.hasInlineContent) {
compareInline(pair.before, pair.after, out) // Only drill down to `words` when needed
}
}
}
// Two key rules:
// 1) A `delete` + `insert` of the same type and sufficiently similar is treated as a `rewrite of the same block`, not two operations;
// 2) Deliberately ignore styles when pairing; after pairing succeeds, separately determine whether the style changed.
There are two tricky issues worth discussing separately, because they are the type where "thinking clearly is harder than writing code."
The first: "rewrite" and "delete + insert" look identical. If you replace a word in a paragraph, from the most naive algorithm's perspective, it's "delete the old paragraph, insert a new paragraph," so the whole paragraph is highlighted again. One treatment is to reassign a "delete + insert" pair of the same type with sufficiently similar content, determine that they are actually "a rewrite of the same paragraph," and then continue drilling in to report only the truly changed words.
The second: changing style should not be read as "delete the plain version, add the bold version." If you bold a word, you expect "this word is now bold," not "delete a word, insert a bold word." So when pairing, deliberately ignore styles, first match the two sides based on "is it the same word," and after pairing succeeds, separately determine whether the style changed. First ask "is it the same thing," then ask "how was it changed" — this order itself is a design.
There is also a coordinate convention that runs throughout: all positions are recorded in the new document, and the old document mainly serves the identity of "contributing deleted content." Moreover, the old and new documents can even come from different versions of the "structure definition"; the algorithm doesn't assume the two sides have exactly the same types — because historical versions often look different from the current one.
4. How to display it: deleted content has no position
After the diff is computed, you get a series of "change points." The next question is how to display it.
In an editor, highlighting usually attaches a layer of "decoration" to the corresponding text. But there's a fundamental problem: deleted content doesn't exist in the new document, so it has no position to attach to.
A straightforward solution: since it can't be attached, redraw the deleted content as a read-only "patch block" inserted at the position where it "used to be." And not as a plain text string, but redrawn in its original form — bold stays bold, deleted table rows are still rows in a table, images are still images. Use the document's own renderer to "serialize" this content, and the deleted content retains its original appearance.
Markup is a layer of "observation" floating above the document, not part of the document itself. It can carry multiple authors' colors and can be removed at any time without touching the original text.
TYPESCRIPT
// Deletions have no position to attach — serialize them into a read-only `patch block` and insert it at the position where it used to be
function decorationFor(doc: Node, change: Change): Decoration {
if (change.kind === 'deleted') {
const patch = serializeWith(doc.schema, change.deletedContent) // Redraw as-is
patch.contentEditable = 'false' // Read-only: it's not content of this version
return widgetAt(change.from, patch) // Attach at its original position
}
return inlineMark(change) // Other changes are just normal highlights
}
Some "unexpected but unavoidable" details illustrate that drawing a diff correctly is itself an engineering task: if the deleted content is an empty paragraph, there's nothing visible on screen, so you need a minimal placeholder; otherwise you won't know that "a paragraph was deleted here." Deleted content should be unselectable for readers, otherwise copying the whole page would also copy characters that are no longer in this version. Block-level elements (images, tables) have no "text" to color, so they can only be faded out as a whole with an underline. Table cells often have their own background colors that would cover the highlight, so you need to draw an extra layer. None of these are the algorithm's fault; they are the inherent weight of "display."
5. The most important protocol: diff is an "observation," not "content"
If this article gets one thing across, it's this.
There is an extremely restrained design decision: all diff marks live only in the editor's "plugin state" and are never written into the document itself.
Because a diff describes "what version this person is currently comparing this version against" — it's an observation, a viewpoint, not content of the document.
Imagine writing it into the document: it would sync to all collaborators, even though others aren't looking at this comparison; it would enter the undo history, so pressing undo might undo a "comparison"; it would be saved as formal content. All three are wrong.
It's more like the highlighter you use on a printed draft: the original text hasn't changed a single character, and that fluorescent layer shouldn't be "photocopied" into the official archive.
TYPESCRIPT
// Diff only goes into `plugin state`, never into the document
apply(tr, prev) {
return {
base, // which version is being compared against
changes: diffDocuments(base, tr.doc), // recompute from scratch on every change, not shift old marks
decorations, // only attach during rendering, not stored in the document
}
}
// The document itself doesn't change a single character: it doesn't participate in sync, doesn't enter the undo stack, isn't saved
This decision triggers a chain of subsequent protocol-level consequences. One is especially elegant: since marks aren't in the document and can be recomputed at any time, the document, on every keystroke, prefers to recompute the diff from scratch rather than "shifting the old marks." Why not shift? Because positions are the conclusion of comparing two documents, not movable labels — changing one character might turn what was originally judged as "whole paragraph rewritten" back into "actually not much changed." This kind of "conclusion changed" can't be expressed by shifting. Recomputing is the only honest approach.
6. Another protocol: for attribution, "who changed it" is not "who wrote it"
Many collaboration scenarios want to know "who made this change." To support that, you need a very strict contract first: the coordinates you give must be coordinates in the new document, and must be derived from that document itself. Why so strict? Because if you parse the same content twice and any normalization rule changes slightly in between, these ranges will silently drift onto someone else — no error, just the wrong name.
What's more worth saying is the stance of "attribution," which can be summarized in one sentence: ranges are an "increment," not a "census."
It should answer "from the old version to this version, who newly added these positions," not "who wrote each current character." What's the difference? If you give a "census," when you change a few characters in an old paragraph and the tool reports it to "this paragraph," it will look up "who created the beginning of this paragraph" — attribution then goes to the person who created this paragraph three years ago, while the person who actually typed is one character away and was never asked. Only an "increment" will fall through on this old paragraph and then naturally ask the person who just typed.
TYPESCRIPT
// Attribution only recognizes `increments`: who newly added what in this version
function authorAt(position: number): string | null {
return ranges.find(r => r.from <= position && position < r.to)?.author ?? null
// If it falls outside all ranges, return null — if unknown, don't attribute
}
// Deletions go through a separate channel: deletion points often sit between old content, so range queries necessarily fall through
The cost is also honestly stated: deletions have almost no attribution. Because deleted content isn't in this version, has no range, and most collaboration systems only record "what was deleted," not "who deleted it." The stance here is unambiguous — if unknown, don't attribute, rather than making up a "last saved person." Because a guessed name and a wrong name are the same thing. There's another case to handle separately: if a whole version is generated at once (e.g., "rollback to previous version"), character-by-character inference would credit the original author, so it's more appropriate to use a "whole-version attribution" — this entire version is the action of the person who pressed the button.
7. Separating the algorithm and the protocol
Going back through: splitting, pairing, drilling, drawing are algorithms; how marks exist, how attribution is judged are protocols. Separating them yields a practical division of labor:
The algorithm is responsible for "calculating accurately, stably, and completely"; the protocol is responsible for "what the result looks like, where the boundaries are, and how it should be interpreted."
What's interesting is that the truly difficult part often isn't the faster loop — the shortest edit script is an old technique that matured decades ago, and the community lacks no implementations. The decisions that are repeatedly weighed are all protocol-level: diff marks must not be written into the document; deleted content must be redrawn; attribution only recognizes increments, and if unknown, don't guess; Chinese must be cut separately; even a requirement like "counting change counts on the server" requires isolating rendering-related code. None of these decisions can be solved by "switching to a faster algorithm" — they all ask the same question: what is this thing, and what identity should it exist as.
So next time you see a tool highlight the whole paragraph red, maybe you'll think: it's not broken, it just hasn't learned how to cut, and hasn't yet figured out "what identity a diff should exist as." And when both of these are figured out, "changing one character" finally marks only that one character.
Top 2 from juejin.cn, machine-translated. The original thread is authoritative.
Combining characters also need to be split at the grapheme level.
The part 'difference is an observation, not content' is what I most agree with. Marks only go into the plugin state, recompute from scratch each time, preferring to compute an extra time rather than shifting old marks—this trade-off is very clean. Shifting looks cheaper, but actually 'the conclusion changed' cannot be expressed. Another point I want to add: 'when tied, always prefer deletion' looks like a detail, but without it, the same pair of documents would produce a different set of markers after each refresh, and the diff tool's credibility would be gone immediately. A question: for tiptap-myers-content-diff, what magnitude of time does a full recompute take on a very large document? If every keystroke triggers a full recompute, would the frequency of budget downgrades be high enough that users often see the coarse result of 'whole paragraph delete + whole paragraph add'?