跪拜 Guibai
← Back to the summary

Stop AI from Copying Legacy Code: Purge Bad Samples First, Then Lock with Types and Tests

Core conclusion: AI is not "learning bad habits"; it treats your codebase as its textbook. The more legacy patterns in the repo, the higher the probability it copies them. So governance works in two steps: first remove the bad examples, take outdated calls, temporary patches, copy-paste blocks, and commented-out dead code out of the "copyable samples"; then push constraints down into the code, using types, encapsulation, tests, and CI gates so that wrong patterns cannot be written, don't compile if written, and fail tests when run. Clean but not lock, and legacy patterns grow back; lock but not clean, and you'll be drowned in existing errors.

Table of Contents

  1. One-line conclusion
  2. Why AI copies old code
  3. Four shapes of bad examples
  4. Why adding rules alone can't stop it
  5. Step 1: Purge the bad examples
  6. Step 2: Seal off wrong patterns with types
  7. Step 3: Seal off retreat paths with tests and gates
  8. Symptom → Cause → Fix table
  9. Runnable legacy scanner script
  10. Why clean first, then lock
  11. Boundaries: when not to be so aggressive

1. One-Line Conclusion

When AI writes code, it mainly relies on two things: common patterns seen in pretraining, and existing patterns in the repo it can read in context. The latter has higher weight because it looks more like the current project's style. So every legacy pattern in the codebase is a demonstration for the model.

The governance order is therefore fixed: first reduce the density of bad examples, then raise the cost of bad patterns. The former addresses "what it copies", the latter addresses "what happens if it copies". Do both, but the order cannot be reversed. Lock first and clean later, and you'll be forced to add a bunch of exemptions for existing errors, effectively turning off the gate with your own hands.

2. Why Models Copy Legacy Code

3. Four Shapes of Bad Examples

Shape What it looks like Why it's dangerous
Deprecated API Calls to replaced functions, e.g., getJSON New patterns only exist in a few new files; legacy patterns are the majority
Temporary Patch # FIXME do this later, next week Patches carry "expired promises" and get copied as a paradigm
Copy-Paste Block The 4th copy of the same logic Fix one, the other three stay wrong, and AI will copy the unfixed one
Commented-Out Code # old_impl = ... It's the strongest legacy example and is not subject to lint

Common trait of all four: the syntax is completely valid. Review can't catch the error; it only breaks at runtime. That's why they are harder to handle than compile errors.

4. Why Prompts Alone Fail

5. Step 1: Purge the Bad Examples

6. Step 2: Seal Off Wrong Patterns with Types

7. Step 3: Seal Off Retreat Paths with Tests and Gates

8. Symptom → Cause → Fix Table

Symptom Cause Fix
Fixed interfaces, AI writes the old pattern next time Old call site density is higher than new pattern Clean existing + funnel to single entry
Every file repeats the same logic block No reusable abstraction Extract common function + duplicate block gate
AI repeatedly writes # TODO later Patches treated as a paradigm Zero tolerance for patches, convert to issue and delete comment
Review can't spot it, breaks at runtime Legacy syntax is valid, just semantically outdated Deprecation list + error message gives alternative
Gate opens and everything is red, forced to bulk ignore Locked before cleaning, existing issues not handled Clean to zero first, then set threshold to zero tolerance

9. Runnable Legacy Scanner Script

The following script does three things in one scan: outdated calls, temporary patch markers, duplicate function bodies. Any hit returns exit code 1, ready to be hooked into CI.

"""Codebase legacy health check: outdated calls / temporary patch markers / duplicate blocks, any hit returns exit code 1."""
import ast, hashlib, pathlib, re, sys

DEPRECATED = {                                    # Legacy pattern → replacement
    "getJSON": "Use request_json()",
    "requests.get": "Go through http_client.fetch()",
}
PATCH = re.compile(r"#.*\b(TODO|FIXME|HACK|TEMP)\b")    # Temporary patch marker


def dotted(node):                                 # Restore call site to dotted full name
    parts = []
    while isinstance(node, ast.Attribute):         # a.b.get() split layer by layer to a
        parts.append(node.attr)
        node = node.value
    if isinstance(node, ast.Name):
        parts.append(node.id)
        return ".".join(reversed(parts))
    return ""


def check(path):                                  # Health check for single file
    src = path.read_text(encoding="utf-8")
    tree = ast.parse(src)
    hits = []
    for node in ast.walk(tree):                    # Traverse all call sites
        if isinstance(node, ast.Call) and dotted(node.func) in DEPRECATED:
            hits.append((path, node.lineno, "outdated call", dotted(node.func)))
    for lineno, line in enumerate(src.splitlines(), 1):
        mark = PATCH.search(line)                  # Find patch markers line by line
        if mark:
            hits.append((path, lineno, "temporary patch", mark.group(1)))
    return hits, tree


def body_key(func):                                # Normalized fingerprint of function body
    body = [ast.dump(n) for n in func.body
            if not (isinstance(n, ast.Expr) and isinstance(n.value, ast.Constant))]
    return hashlib.md5("".join(body).encode()).hexdigest()


def main(root):                                    # Aggregate report and decide exit code
    files = sorted(p for p in pathlib.Path(root).rglob("*.py")
                   if ".venv" not in p.parts)
    all_hits, seen, dup = [], {}, []
    for path in files:
        hits, tree = check(path)
        all_hits += hits
        for node in ast.walk(tree):                 # Same body function as copy-paste
            if isinstance(node, ast.FunctionDef) and len(node.body) >= 3:
                key = body_key(node)
                if key in seen:
                    dup.append((seen[key], f"{path.as_posix()}:{node.name}"))
                else:
                    seen[key] = f"{path.as_posix()}:{node.name}"
    for path, lineno, kind, detail in all_hits:
        print(f"{path.as_posix()}:{lineno}  [{kind}] {detail}")
    for left, right in dup:
        print(f"duplicate function body  {left}  ==  {right}")
    bad = len(all_hits) + len(dup)
    print(f"\nScanned {len(files)} files, outdated/patch {len(all_hits)} places, duplicate blocks {len(dup)} groups")
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "."))

Run python scan_legacy.py app on a small 4-file directory (app/user.py has 2 outdated calls and 1 FIXME, app/report.py has 1 HACK, app/order.py and app/user.py each have a same-name same-body normalize), actual output is:

app/report.py:1  [temporary patch] HACK
app/user.py:5  [outdated call] requests.get
app/user.py:6  [outdated call] getJSON
app/user.py:5  [temporary patch] FIXME
duplicate function body  app/order.py:normalize  ==  app/user.py:normalize

Scanned 4 files, outdated/patch 4 places, duplicate blocks 1 group

Exit code 1, which is a failure in CI. Running the same script on a directory with only clean modules outputs "Scanned 1 file, outdated/patch 0 places, duplicate blocks 0 groups", exit code 0.

Limitations should be clear: duplicate detection uses function body fingerprints; semantically equivalent but differently written logic after refactoring isn't recognized. For production, you can switch to jscpd or pylint's duplicate-code check; the idea is the same, with higher precision.

10. Order Matters

Most teams go straight to lint and type checking without cleaning old code. The first run shows 40 old calls all red. At that point there are only two options: fix all (which is cleaning first, just forced to do it all at once), or add ignores. Most choose the latter.

Every ignore added exempts AI one more time when it writes the old pattern. 40 ignores are 40 passes, the gate degrades from "rejection" to "reminder", and you just said in §4 that reminders can't stop the model.

The economics behind order: cleaning is a one-time cost, gates are an ongoing cost. Clean first, and you can set the threshold to zero tolerance. Lock first, and you can only set "not worse than yesterday", which has no binding force on the model because it doesn't look at yesterday.

When the existing volume is too large, a compromise is to set decreasing quotas per directory: zero tolerance for new code, old directories multiply the quota by 0.9 per week, starting from 40, after 12 weeks it drops to about 11 (40 * 0.9^12 ≈ 11.3). This is the only way to keep both gate strength and landing speed.

11. Boundaries: When Not to Be So Aggressive

Only when the repo has more correct patterns than legacy ones will AI reliably copy correctly. So clean first, then lock. Doing these two actions a week apart is fine, but if the order is reversed, you'll have to start over.

This column "AI Development Paradigm" focuses on code organization, code review, and engineering constraints under AI collaboration, from directory structure to type design and CI gates. Each article includes a runnable check script and updates weekly. To read in order, subscribe on the column page.