AI Skills Need a Test Harness, Not Just a Demo Run
Most AI Skills ship after a single successful run and degrade silently when colleagues use different phrasing or when model behavior shifts. A small, repeatable test harness that reads the live SKILL.md catches undertriggering, hallucinated facts, and structural drift before they reach a teammate, and it costs a single Python file.
A Skill that works once on a clean prompt is a demo; a Skill that survives messy real-world input and repeated runs is infrastructure. The difference is a test harness that checks three things: whether the Skill triggers on the right requests and stays quiet on the wrong ones, whether its output meets explicit success criteria without hallucinating missing details, and whether its structure holds across five consecutive runs. The approach starts with a single hard sample — the messiest meeting notes available — and iterates the Skill until it passes, then expands to more cases. A lightweight Python script reads the actual SKILL.md, sends the description to the model for trigger tests, and runs functional and consistency checks against the full instructions. It is not a production Eval framework, but it catches regressions in seconds and points directly to the failing layer: description, instructions, or structural drift.
Skill testing is essentially a three-layer contract: the description is a routing contract, the instructions are a behavior contract, and repeated runs verify the contract holds under non-determinism.
Hallucination in Skills often manifests not as obvious nonsense but as confidently filling in plausible details the source never provided — a meeting date, an owner, a decision — which makes it harder to spot without explicit negative assertions in tests.
The advice to start with one hard sample rather than twenty easy ones inverts the typical test-first instinct and aligns better with how prompt engineering actually iterates: fix the worst failure, then generalize.
Reading the live SKILL.md rather than a copied version eliminates the most common test-maintenance failure mode in AI tooling: tests that pass against a stale copy while the real Skill has already diverged.