Codex's Test-Retry Token Hole: Prompts and Rules to Stop the Loop
Token costs and wasted time are the price of letting AI agents guess at failures. These concrete prompts give developers a way to force Codex to show evidence and stop when it's spinning, without losing the safety of a test pass.
Lao Jin watched Codex loop through failing tests, each round modifying and retesting while token usage climbed. Checking the token bill revealed the cause: the agent retries the same failing checks without new evidence. He collected prompts that force a pause, demand a review of existing logs and token usage, and limit each round to the specific fix and root cause. The four-step stop rule: after two failed attempts at the same issue, pause and report.
The workflow also covers verifying a claimed fix by redoing the original action, not just trusting the test pass, and refusing to delete failing tests. For long-term control, he appends rules to AGENTS.md that encode these constraints: re-use valid checks, don't re-verify unchanged data, stop after two failures, and keep brief task records for resumption. Anthropic's internal review process saw the share of reviews needing five or more rounds drop from 8.9% to 1.5% after similar rules.
Agents lack reliable memory, so they re-verify unchanged data; explicit prompts that force them to cite existing evidence are more effective than hoping they'll behave sensibly.
Token consumption is a hidden cost of agent autonomy; a developer's patience is being monetized by the retry loop.
The stop-after-two rule treats a failed fix as a signal to re-think, not to keep hammering, which aligns with how a human debugger would behave.