Stop AI from Copying Legacy Code: Purge Bad Samples First, Then Lock with Types and Tests
Core conclusion: AI is not "learning bad habits"; it treats your codebase as its textbook. The more legacy patterns in the repo, the higher the probability it copies them. So governance works in two steps: first remove the bad examples, take outdated calls, temporary patches, copy-paste blocks, and commented-out dead code out of the "copyable samples"; then push constraints down into the code, using types, encapsulation, tests, and CI gates so that wrong patterns cannot be written, don't compile if written, and fail tests when run. Clean but not lock, and legacy patterns grow back; lock but not clean, and you'll be drowned in existing errors.
Table of Contents
- One-line conclusion
- Why AI copies old code
- Four shapes of bad examples
- Why adding rules alone can't stop it
- Step 1: Purge the bad examples
- Step 2: Seal off wrong patterns with types
- Step 3: Seal off retreat paths with tests and gates
- Symptom → Cause → Fix table
- Runnable legacy scanner script
- Why clean first, then lock
- Boundaries: when not to be so aggressive
1. One-Line Conclusion
When AI writes code, it mainly relies on two things: common patterns seen in pretraining, and existing patterns in the repo it can read in context. The latter has higher weight because it looks more like the current project's style. So every legacy pattern in the codebase is a demonstration for the model.
The governance order is therefore fixed: first reduce the density of bad examples, then raise the cost of bad patterns. The former addresses "what it copies", the latter addresses "what happens if it copies". Do both, but the order cannot be reversed. Lock first and clean later, and you'll be forced to add a bunch of exemptions for existing errors, effectively turning off the gate with your own hands.
2. Why Models Copy Legacy Code
- The training objective is "the most likely next token", not "the most correct". Imitation is its default behavior, not a bug or laziness.
- Examples in context are weighted by "recent + similar". If
getJSONappeared on line 40 of the same file, the probability of writing it again on line 400 is much higher than the probability of stopping to check documentation. - Quantity distribution determines the outcome: suppose there are 30 legacy patterns and 3 new patterns in the repo, the model's naive probability of copying the legacy pattern is 30/33 ≈ 0.909. That's not the model being dumb; that's the sample distribution you provided.
- A more subtle point: commented-out dead code also counts in context.
# old = requests.get(...)looks the same as live code to the model, both are evidence that "this is how this project is written". - Analogy: the spec document you write is the "constitution", the 300 old calls in the repo are "case law", and AI follows a case-law system. It looks at case law first, and only checks the constitution if it really can't find anything.
3. Four Shapes of Bad Examples
| Shape | What it looks like | Why it's dangerous |
|---|---|---|
| Deprecated API | Calls to replaced functions, e.g., getJSON |
New patterns only exist in a few new files; legacy patterns are the majority |
| Temporary Patch | # FIXME do this later, next week |
Patches carry "expired promises" and get copied as a paradigm |
| Copy-Paste Block | The 4th copy of the same logic | Fix one, the other three stay wrong, and AI will copy the unfixed one |
| Commented-Out Code | # old_impl = ... |
It's the strongest legacy example and is not subject to lint |
Common trait of all four: the syntax is completely valid. Review can't catch the error; it only breaks at runtime. That's why they are harder to handle than compile errors.
4. Why Prompts Alone Fail
- Rules are soft constraints: written in prompts, rules files, or READMEs, they only change probability, not feasibility. The model can read "don't use getJSON" and write it anyway.
- Rules dilute each other: the more irrelevant rules you hang, the more the key one gets drowned. Constraints in code don't dilute; the 41st lint rule works like the 1st, hitting it reports an error.
- In 项目越写越乱别急着改架构 we said: quantify the directory mess before acting. Cleaning bad examples is the same, count first, then delete, not by feeling.
- One sentence distinction: rules are reminders, compilers and tests are rejections. Reminders can be ignored, rejections cannot.
5. Step 1: Purge the Bad Examples
- Count first, then delete. Run a health check (the script in §9 works) to get three lists: outdated calls, patch markers, duplicate blocks. Without a list, acting usually results in deleting half and the other half is even messier.
- Sort by spread speed, not file order. The old API with the most references goes first, because every copy doubles its cost.
- One entry replaces N call sites. Wrap all old calls into one encapsulating function, then replace file by file. Each file replacement is a separately revertable commit.
- Patches must not be left ownerless. Either fix them on the spot, or convert to issues and delete the comment. A dated FIXME is better than an ownerless FIXME, but zero is best.
- Dead code: delete directly. Git has history; comments provide no extra information but keep polluting the context.
- Clean sample code and docs together. Examples in README and docs are the strongest samples; their weight in the model's eyes is higher than business code.
6. Step 2: Seal Off Wrong Patterns with Types
- Use enums instead of strings. Change
mode: strtoLiteral["fast", "safe"]or an Enum; old values fail to compile. If AI tries to writemode="legacy", type checking blocks it first. - Use parameter objects instead of scattered parameters. Change
create(a, b, c, d)tocreate(cfg: CreateConfig), so old positional parameter patterns don't match. - Tighten return types. Replace
dictwith dataclass; if AI tries to shove in guessed keys likepayload["data"], it will report that the attribute doesn't exist. - Privatize plus single entry point. Make
_sessionprivate inside the module, expose onlyquery_one(). To bypass the encapsulation and execute SQL directly, it must explicitly modify a private member, which is enough to make it hesitate. - Analogy: the type system is a mold, not a manual. You can ignore a manual, but a mold won't produce the part if it doesn't fit.
7. Step 3: Seal Off Retreat Paths with Tests and Gates
- Write a failing test for the new pattern (red first), then change the implementation (green). This test becomes a mousetrap: if anyone copies the old pattern later, CI immediately turns red.
- Gates have three layers: static checks (type / lint) → unit tests → custom rules (§9 script). The first two run locally for speed; all three run in CI to ensure enforcement.
- Error messages should guide modification. Don't just say "calling getJSON is forbidden", say "don't use getJSON, use request_json()". The model reads the error and follows it; if it reads
forbidden, it just finds another way around. - Don't create an archive directory for bad examples. Moving old implementations to
legacy/and exempting them is keeping an "officially recommended copyable sample" in the repo. This is counterintuitive; the reason is in §10. - The value of gates is not blocking people, but blocking probability. A person understands at a glance; the model re-reads the context every time.
8. Symptom → Cause → Fix Table
| Symptom | Cause | Fix |
|---|---|---|
| Fixed interfaces, AI writes the old pattern next time | Old call site density is higher than new pattern | Clean existing + funnel to single entry |
| Every file repeats the same logic block | No reusable abstraction | Extract common function + duplicate block gate |
AI repeatedly writes # TODO later |
Patches treated as a paradigm | Zero tolerance for patches, convert to issue and delete comment |
| Review can't spot it, breaks at runtime | Legacy syntax is valid, just semantically outdated | Deprecation list + error message gives alternative |
| Gate opens and everything is red, forced to bulk ignore | Locked before cleaning, existing issues not handled | Clean to zero first, then set threshold to zero tolerance |
9. Runnable Legacy Scanner Script
The following script does three things in one scan: outdated calls, temporary patch markers, duplicate function bodies. Any hit returns exit code 1, ready to be hooked into CI.
"""Codebase legacy health check: outdated calls / temporary patch markers / duplicate blocks, any hit returns exit code 1."""
import ast, hashlib, pathlib, re, sys
DEPRECATED = { # Legacy pattern → replacement
"getJSON": "Use request_json()",
"requests.get": "Go through http_client.fetch()",
}
PATCH = re.compile(r"#.*\b(TODO|FIXME|HACK|TEMP)\b") # Temporary patch marker
def dotted(node): # Restore call site to dotted full name
parts = []
while isinstance(node, ast.Attribute): # a.b.get() split layer by layer to a
parts.append(node.attr)
node = node.value
if isinstance(node, ast.Name):
parts.append(node.id)
return ".".join(reversed(parts))
return ""
def check(path): # Health check for single file
src = path.read_text(encoding="utf-8")
tree = ast.parse(src)
hits = []
for node in ast.walk(tree): # Traverse all call sites
if isinstance(node, ast.Call) and dotted(node.func) in DEPRECATED:
hits.append((path, node.lineno, "outdated call", dotted(node.func)))
for lineno, line in enumerate(src.splitlines(), 1):
mark = PATCH.search(line) # Find patch markers line by line
if mark:
hits.append((path, lineno, "temporary patch", mark.group(1)))
return hits, tree
def body_key(func): # Normalized fingerprint of function body
body = [ast.dump(n) for n in func.body
if not (isinstance(n, ast.Expr) and isinstance(n.value, ast.Constant))]
return hashlib.md5("".join(body).encode()).hexdigest()
def main(root): # Aggregate report and decide exit code
files = sorted(p for p in pathlib.Path(root).rglob("*.py")
if ".venv" not in p.parts)
all_hits, seen, dup = [], {}, []
for path in files:
hits, tree = check(path)
all_hits += hits
for node in ast.walk(tree): # Same body function as copy-paste
if isinstance(node, ast.FunctionDef) and len(node.body) >= 3:
key = body_key(node)
if key in seen:
dup.append((seen[key], f"{path.as_posix()}:{node.name}"))
else:
seen[key] = f"{path.as_posix()}:{node.name}"
for path, lineno, kind, detail in all_hits:
print(f"{path.as_posix()}:{lineno} [{kind}] {detail}")
for left, right in dup:
print(f"duplicate function body {left} == {right}")
bad = len(all_hits) + len(dup)
print(f"\nScanned {len(files)} files, outdated/patch {len(all_hits)} places, duplicate blocks {len(dup)} groups")
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "."))
Run python scan_legacy.py app on a small 4-file directory (app/user.py has 2 outdated calls and 1 FIXME, app/report.py has 1 HACK, app/order.py and app/user.py each have a same-name same-body normalize), actual output is:
app/report.py:1 [temporary patch] HACK
app/user.py:5 [outdated call] requests.get
app/user.py:6 [outdated call] getJSON
app/user.py:5 [temporary patch] FIXME
duplicate function body app/order.py:normalize == app/user.py:normalize
Scanned 4 files, outdated/patch 4 places, duplicate blocks 1 group
Exit code 1, which is a failure in CI. Running the same script on a directory with only clean modules outputs "Scanned 1 file, outdated/patch 0 places, duplicate blocks 0 groups", exit code 0.
Limitations should be clear: duplicate detection uses function body fingerprints; semantically equivalent but differently written logic after refactoring isn't recognized. For production, you can switch to jscpd or pylint's duplicate-code check; the idea is the same, with higher precision.
10. Order Matters
Most teams go straight to lint and type checking without cleaning old code. The first run shows 40 old calls all red. At that point there are only two options: fix all (which is cleaning first, just forced to do it all at once), or add ignores. Most choose the latter.
Every ignore added exempts AI one more time when it writes the old pattern. 40 ignores are 40 passes, the gate degrades from "rejection" to "reminder", and you just said in §4 that reminders can't stop the model.
The economics behind order: cleaning is a one-time cost, gates are an ongoing cost. Clean first, and you can set the threshold to zero tolerance. Lock first, and you can only set "not worse than yesterday", which has no binding force on the model because it doesn't look at yesterday.
When the existing volume is too large, a compromise is to set decreasing quotas per directory: zero tolerance for new code, old directories multiply the quota by 0.9 per week, starting from 40, after 12 weeks it drops to about 11 (40 * 0.9^12 ≈ 11.3). This is the only way to keep both gate strength and landing speed.
11. Boundaries: When Not to Be So Aggressive
- One-off scripts, prototypes, temporary data cleaning: no need for this system, the constraint cost exceeds the benefit.
- Third-party SDKs that can't be migrated: don't force-delete calls, isolate with an encapsulation layer; the gate forbids "direct calls" not "calls".
- Repos under 5k lines, teams of 3 or fewer: only do the health check script, make the existing volume visible first.
- Don't make it an annual cleanup. Clean once, and it grows back after three months. Effective is incremental locking: zero tolerance for new code, decreasing quotas for existing volume, making "bad example density" monotonically decrease.
Only when the repo has more correct patterns than legacy ones will AI reliably copy correctly. So clean first, then lock. Doing these two actions a week apart is fine, but if the order is reversed, you'll have to start over.
This column "AI Development Paradigm" focuses on code organization, code review, and engineering constraints under AI collaboration, from directory structure to type design and CI gates. Each article includes a runnable check script and updates weekly. To read in order, subscribe on the column page.