Codex's Test-Retry Token Hole: Prompts and Rules to Stop the Loop
Recently, while using Codex, Lao Jin kept waiting.
Tests failed, it modified the code, tested again, still failed, then it modified again.
He watched it work, thinking this round would finally finish, but it just kept going.
It really never failed to keep trying - it tried, and he got annoyed.
The wait got too long, something felt wrong, so he made it count token usage to see where everything went.
After seeing the result, he just wanted to curse.
The back-and-forth of analyzing and modifying had already eaten a huge number of tokens. He waited patiently, it consumed patiently - neither's patience was wasted.
This isn't an ordinary pit, it's a giant pit Pro Max.
But to just skip testing and not check? He didn't dare. If saving checks results in handing me something unusable, that's an even bigger pit.
The worst case after waiting: it says "Passed 237,272 tests ✅", then you click run, and get "Error xxx, cannot start".
He cursed (censored).
These past couple of days Lao Jin went through related discussions to see if others had the same experience and how they resolved it.
For example, this guy also complained: a simple task generated a pile of tests - is my quota all being spent here?
Another brother said: changing a small feature stuffed in a bunch of checks and tests.
There was a guy who used both Codex and Claude, and he stopped playing the guessing game with it.
His approach was: first reproduce the problem, look at the error, see what changed recently, find which step started going wrong, then make the minimal change and check the result.
This one we can follow step by step: first let the AI shut up and stop rambling, then ask it what it plans to check in the next round.
First make it stop guessing and see what's already in hand
Suppose you ask the AI to build a small file-organizing tool, and clicking export does nothing, so you ask it to fix it.
It changed something, the file is there, but you open it and find it's empty, so you need it to look at where the content failed to be written.
Send the following into the current conversation to have it first review the existing records.
Don't just ask what it did - answering itself would make it do it again, like losing your coat and your pants.
Get the reply, then see what's missing, what it found, and what it plans to do next.
This information is what we use to make a judgment.
# Pause first, preserve the scene
Safely pause subsequent modifications and reruns; don't forcibly interrupt ongoing data writes.
Only read existing records; explain the recent failure, tried approaches, new findings, and their basis. If you don't know the reason, say so directly.
# Separate known from unknown
List the commands and results that already passed checks and their applicable states; don't rerun to fill records.
When relevant code, dependencies, config, test data, or environment changed, mark what needs re-verification.
Based on the error and recent changes, propose only one reasoned reproduction, diagnostic, or fix action. If materials are missing, list them; don't execute yet.
# Report usage as actuals
If you have records, report the source, time range, and token count. If only totals exist, report the total.
Without records, don't guess step-by-step consumption, cost, or plan balance.
If it says it's missing a sample file, give it the file and explain where it's placed.
If you haven't uploaded it yet and it's already searched several rounds, even more searching won't conjure the file from nothing.
If an external service it needs isn't reachable, handle the connection first. If it can't be restored, stop and wait until materials and connections are ready, then let it see if that step can proceed.
If it finds that the export got empty data and plans to trace backward to see where the content was lost, that's fine to continue.
Not finding the answer is fine; you just need to know where to look next.
The worst is cycling through several old methods and testing against the same error - then it gets really messy. You don't see practice making perfect; you see practice making consumption.
Anthropic, in their internal code review, specifically handled this problem of repeated fixing and repeated checking. They used several approaches to prevent going back and forth: after three rounds, step back and rethink the plan; after the first few rounds, focus review on new changes; handle similar errors together; don't repeatedly bring up already-mentioned issues.
The team reported that after these adjustments, the proportion of reviews needing five or more rounds dropped from 8.9% to 1.5%.
Vox then organized a similar set of four requirements for AI, one of which was more direct: if the same problem isn't fixed after two attempts, stop and report first. We can also set two attempts as our own stopping point: first see what was tried and where you're stuck, then decide how to continue.
If it hasn't been fixed after two attempts, stop and think differently—don't turn failure into passing.
If it says this check passed before and it just wants to re-confirm, tell it to find the record.
If the code hasn't changed and the run conditions haven't changed, you don't have to retake the exam with it just because it forgot.
But if the relevant code or run conditions changed, make it explain which checks are affected and which need to be added.
Once you know the next step, let it act
Once you know the next step, you also need to see how big a change it plans to make.
Going back to the export problem: you ask it to fix it, but it plans to redo the page, swap out the file processing, and add a few options that might be useful later. Ask why - "just in case".
It hasn't done the first thing yet, but it's already arranged for all the maybes.
Afraid you won't want to extend later, it adds layers. Afraid you won't have enough features later, it adds items. It sees the future loud and clear, but your file still isn't out.
First have it remove these future-oriented tasks and return to this export. Unrelated old problems can be listed; don't fix everything you see as you go.
Under that post asking whether tests were eating the quota, a guy said a simple migration task got bigger and bigger, until he took it back and guided it step by step himself to finish.
The OpenCode team also put this requirement into their project rules: to add complexity for hypothetical problems, first explain the specific problem and cost, and ask for consent. If an existing implementation works, keep using it; fixing a feature doesn't mean starting from the Big Bang.
Before modifying, you can also make it search within the relevant code to see if the same cause has caused problems elsewhere. If it's confirmed to be the same root cause and within this round's scope, handle it together; don't patch one here and claim another one next time.
Once it's clear what to change, then see how to check. If only the export is changed, check the export first. If it changed a place that other features also use, check the affected features together, and complete the project-required checks when handing off.
Gemini CLI's own project rule keeps the heavy checks like cleanup, install, build, and test for the end of the code implementation task. If errors occur in the middle, first use targeted checks to find the cause; don't mobilize the whole family for every change, but don't save everything until the end either.
If a check already passed and the result is still valid, don't redo it. Do the checks that need adding, and don't turn saved effort into trouble for the user.
Once what to check, what to change, and how to check all line up, send these four items. If it circles back to the same problem, stop per the agreement, without waiting for you to shout stop. When done, hand over actual results and acceptance steps, not just a grander plan for next steps.
# 1. Only change issues related to this round
Continue per the confirmed plan, keep a recoverable version, reuse existing implementations.
Unrelated old problems: only list them, don't fix or add features opportunistically. Expanding scope first ask me.
For real user data, destructive operations, or extra paid costs, confirm first.
# 2. First find the same root cause
Use the error and existing evidence to locate, don't modify based on guesses.
Search within this round's relevance for same-root, same-pattern issues; after confirming, do minimal fixes together; don't replace everything.
# 3. Each round review affected parts
Focus on reviewing this round's changes; read related context and call chains as needed.
Already-resolved and unaffected issues don't get repeated; unresolved ones continue to be tracked.
Reuse valid checks; complete affected-function and handoff checks per project requirements.
Don't delete failing tests, don't skip necessary checks, don't loosen pass criteria, don't hide errors.
When done, state changes, actual check results, unconfirmed items, and give me acceptance steps.
# 4. If the same problem isn't fixed after two attempts, stop
Safely pause subsequent modifications and reruns; don't forcibly interrupt ongoing data writes.
Report tried methods, actual changes, remaining failures, and items needing my decision; wait for my confirmation before continuing.
If it fails the same way with no new evidence, stop before the two-attempt limit.
After it says it's fixed, do the thing that didn't work again
When it says it's fixed, click export once, download, open, and see if the content is what you wanted, and whether formatting and such are fine.
Also try one or two previously commonly used functions, just in case you get a buy-one-get-one: export fixed, upload broken.
Google's testing materials mentioned a case: some checks are very fine-grained; if the code's internal implementation changes, they immediately report an error. But whether the result is correct, they don't care.
For this export tool, what we want is that the content that should be exported really lands in the file.
Be careful not to delete whichever test reports an error. You're asking it to solve a problem, not to delete the system.
Another guy complained: the search actually failed, but the program returned "no search results".
You think your keywords weren't good, so you change them, search again, still nothing, change again - the software has a bug, but at least it trained you into a thesaurus.
If it can be recovered, recover it; if not, make the failure clear, and then you can make a judgment.
Next time you start, don't let it start from scratch again
This time it's done; you'll reuse it later. Otherwise, when you switch projects, it gets to know the project all over again, and you have to retell your whole life story.
People in the community leave records for long tasks, separating goals, check evidence, and whether something was completed.
He later reported that forgetting the original goal and treating simulated tests as real completions happened less often, but the AI still kept adding designs and tests.
Its memory got better, but its hands are still too active, active enough to keep finding things to do when it should stop.
Don't make records into long memoirs; leave the goal, check results, unresolved issues, and next steps, and read them first before resuming a task.
Then there's another case: data that hasn't changed gets checked again right after being checked.
For simple logic used only once, it first breaks it into many parts, looks detailed now, but the big pit is that next time you change it, you have to babysit every piece.
This guy shared a few rules: avoid repeated checks, don't over-complicate simple things too early.
Of course, new external data still needs checking. We're telling it not to do redundant work, not to skip the necessary ones.
The four prompts above let it continue processing this fix; the following one is for setting default rules for the future, already incorporating those four. You can copy it whole to Codex and have it merge into the project's active AGENTS.md - that's the file of work requirements for AI.
After merging, start a new session, have it confirm it read it, and then start your original task. But just because it will follow this from now on doesn't mean it should now do a whole-project cleanup.
# This round only updates the rules file
Incrementally merge the following rules into the project's currently active AGENTS.md, keep existing rules, ask if conflicts arise, don't overwrite the whole file.
Only change the rules file; don't modify project code based on this. When done, report the file location and changes.
# Scope and implementation
First find existing implementations, only do this round's task, list unrelated old problems only.
Adding dependencies, changing approaches, expanding scope, or adding complexity for hypothetical needs: confirm first.
Before acting, check within relevant scope for same-root, same-pattern issues; after confirming, do minimal fixes together; don't replace everything.
Simple logic used only once: don't abstract prematurely. Split only when there's real reuse, complex detail, or independent responsibility.
Data already verified and unchanged: don't re-verify. Still verify across new trust boundaries; keep permission checks; type assertions can't replace verification.
# Errors and checks
Caught errors must either recover, add context then rethrow, or clearly respond as failed. Preserve necessary cleanup; don't fake success.
Test according to required behavior; reuse valid results; keep necessary checks for different risks and real flows.
Each round focus review on new changes; read relevant context as needed; don't bring up already-resolved and unaffected issues; keep tracking unresolved ones.
Don't delete tests, skip necessary checks, loosen criteria, or hide errors just to pass.
If the same problem fails twice after fixes, safely pause and report tried methods, changes, remaining failures, and pending decisions; continue only after confirmation.
Same failure with no new evidence can stop early; don't forcibly interrupt data writes.
# Long-task resumption
Briefly record goal and acceptance, checked commands and results with applicable states, unresolved items, and next steps. Read before resuming.
When relevant code, dependencies, config, test data, or environment changes, re-evaluate verification scope. Passing mock tests doesn't count as accepted real flow.
If you use these practices repeatedly, you can also organize them into a Skill.
Lao Jin's Kim_Service has a meta-skill-creator for creating and improving skills. Leave common steps behind, so you don't have to start from scratch every time.
The address is on GitHub: search Kim_Service. Usage is simple; just say it in plain language, Use meta skill creator, help me create XXXXX, and tell me how to trigger it.
Final words
Previously, watching the AI keep running, it was easy to think: wait a bit longer, maybe the next round will work.
After this research, Lao Jin thinks: waiting is okay, but you should first know what it plans to do.
If it's still spinning among the original guesses, you need to pull out the records first.
With enough information, we can judge what to do next.
Instead of repeatedly testing the same thing inside that framework of AI's poor memory.
Thank you for reading my article.
If you liked it, a like, a share, and a repost would be nice 🙂
If you want to receive updates as soon as they're out, give me a star ⭐ - thanks for reading.
Open-source knowledge base (real-time updates, discussion groups):
https://my.feishu.cn/wiki/OhQ8wqntFihcI1kWVDlcNdpznFf