A Solo Developer's Loop Engineering System That Learns From Its Own Mistakes

Loop Engineering: A Half-Year Hands-On Breakdown — The Self-Evolving Development System Is Now Open Source

Opening — A Buzzword Caught Up with Me

In June 2026, Loop Engineering suddenly exploded across Silicon Valley AI circles.

First, Boris Cherny (head of Claude Code) said: "I basically don't prompt Claude directly anymore. I write loops — let the loop prompt Claude, then decide the next step on its own." Then Google Cloud's Addy Osmani gave the idea a formal name. After that, Chinese media like 36Kr, TMTPOST, and ZhiDongXi all jumped on the bandwagon.

As I read these articles, it all felt increasingly familiar.

Multi-agent collaboration, maker-checker architecture, structured development pipelines, sub-agent isolation — I suddenly realized that what I'd been running for nearly half a year finally had a term to describe it.

There's already plenty of conceptual discussion online. So in this article, I'm going to directly deconstruct this loop engineering system using my own practice — it has been running in a real production environment for half a year. Every design decision has real usage traces behind it, including the parts that turned out to be useless.

The entire system has been open-sourced as the claude-ship repository. Clone it, run a single install.sh, and it installs into ~/.claude. After that, you can run /clarify → /architect → /ship → /retro in any project.

Section 1: Three Fatal Weaknesses of Coding Alone

Over half a year ago, I was like most people doing AI-assisted programming. The starting point was vibe coding — describe requirements in natural language, AI generates code, I review, test, and fine-tune. The efficiency gains were real: complex features went from days to hours, prototype validation from weeks to tens of minutes.

But the efficiency gains also brought three new problems:

First, confirmation bias. When you converse with the same AI model, it follows your train of thought. You say "this design should use approach A," it says "approach A is great, here's why." You'll never know if approach B is better — because no one challenges you. A one-person brainstorm is essentially an echo chamber.

Second, review fatigue. Initially, you're full of vigilance toward AI-generated code, reading every line. Two weeks later, you start skimming. A month later, you only look at the diff. Three months later, you only glance at key functions. This isn't laziness — it's the cognitive economics of the human brain: if 95% of AI-generated code is correct, your brain automatically reduces the review budget to 5%. The problem is, that 5% of errors don't label themselves.

Third, memory evaporation. How did you fix that weird bug three months ago? Did you write a comment at the time? The comment says "fix edge case" — that's useless to you three months later. A one-person team has no colleague to ask "do you remember that issue?" Every forgotten detail is a re-debugging session.

These three problems are fundamentally human problems. AI just faithfully exposes weaknesses that have always existed in solo development but were previously masked by team structures.

So the real engineering question is: Can we use a set of systematic constraints to cage these problems?

That's the starting point of this loop.

Section 2: The Seven-Step Pipeline — Not Smarter, Just Less Stupid

Let's start with the overall structure.

Seven agents, each with an independent personality definition, tool permissions, and model assignment. Outputs uniformly land in the <feature-name>/ directory, forming a complete "seven-piece set": requirements → design → third_party_review → implementation → review → test_report → retro.

Model assignment for each agent isn't random: review uses Opus (high judgment, slow but accurate), dev and qa use Sonnet (fast execution, cheap), clarify and architect use the default model. This isn't "use whichever is stronger" — it's "what cognitive characteristics does the task require."

If you're curious about the specific implementation, the repository structure is straightforward — commands/ is the slash command entry point, agents/ holds agent personality definitions, templates/development/ contains eight document templates, scripts/third-party-review.sh is a headless Claude Code wrapper for cross-vendor review. A single install.sh installs everything into ~/.claude.

I won't go through each agent one by one — that would turn into a manual. The agent definitions themselves are all in the repository's agents/ directory, plain text, takes five minutes to read through. I'll pick a few design decisions most worth expanding on — four at the agent layer in this section, and the orchestration layer gets its own section.

Decision One: Ban "By the Way" — Attention Is Serial, So Should Questions Be

Most requirement clarification approaches produce a checklist: "Please answer the following 5 questions." Highly efficient, but low quality — because human attention is serial; by the time you're answering question 3, you're already thinking "when will this end."

My clarify agent has a hard constraint: Ask only one question at a time. Batch questioning is forbidden. "By the way" is forbidden.

And it reads the code first before asking. It doesn't ask "how does your code do this" — it asks with code context. If a question can be answered directly from the code, it shouldn't appear in the conversation.

This design was forced by real experience. Early on, I had AI help clarify requirements, and it would dump seven or eight questions at once — I'd answer the first three and completely forget the rest. After switching to single-question loops, the depth of each conversation round noticeably increased, and the resulting requirements.md became more specific and actionable.

The cost is speed. Clarifying one feature may take 4-6 rounds of conversation. But there's an exit mechanism: the user can input "enough" / "start design" at any time to immediately terminate the loop and produce the document based on current information. If it reaches round 8 without finishing, the system proactively pauses questioning — preventing infinite inquiry.

Decision Two: Predict Before Reading — Code Review That Aligns with Cognitive Science

This is the design I'm most satisfied with in the entire system.

The review agent's workflow isn't "read code → find problems." It is:

Don't read the code first. Read only design.md and implementation.md, and based on the design, list 3-5 most likely defect areas (e.g., "error handling might be incomplete," "concurrency scenarios might have race conditions").
Then review the code with these predictions in hand.
Record problems that the predictions hit, and also record problems the predictions missed — the latter is more important, indicating blind spots in the prediction.

I didn't come up with this on a whim. There's a classic finding in cognitive science: if you see the answer first and then give reasoning, you overestimate your reasoning ability. Conversely, if you give predictions first and then see the answer, calibration accuracy improves dramatically. The review's "pre-commit prediction" is the engineering of this finding.

Going further, the starting point for predictions isn't the reviewer's intuition — it's memory. Every time a feature is completed, the retro agent stores hard-won lessons into memory. When the review for the next feature starts, it first retrieves relevant historical incident patterns from memory as candidate starting points for predictions. This means the system's review capability grows with usage count.

Decision Three: Tiered Blocking + Low-Risk Fix — No Back Door for "Deal With It Later"

After review finds problems, they aren't all thrown at dev to fix. Tiered:

Critical / Important: Block the loop, must fix or obtain whitelist exemption
Minor / Suggestion: Apply the "low-risk fix" rule

The key is this rule: for Minor / Suggestion issues, if four conditions are simultaneously met — objective basis, no side effects (no changes to public API/data format/external behavior), change volume ≤20 lines and single file, no user confirmation needed — must fix. Saying "this isn't important, skip for now" is not allowed.

This solves a real problem: reviewer raises a bunch of minor suggestions, dev thinks they're "unimportant" and skips them all. Next feature, same suggestions raised, skipped again. Three months later, 100 minor issues become an unmaintainable codebase. The essence of "low-risk fix" is trading low-friction execution for zero technical debt accumulation.

Meanwhile, deferral reasons for Important issues have a whitelist — only four legitimate reasons: out of scope, requires user confirmation, requires major refactoring, conflicts with design. Vague language like "optimize later" or "temporarily ignore" is forbidden. Reviewer seeing such language marks it as rejected-defer, still blocking the loop.

Decision Four: Not All Experience Is Worth Storing — Three Iron Rules Filter Out Nonsense

Most attempts at "summarizing lessons learned" end up as collections of platitudes: "write tests," "watch out for concurrency," "documentation is important" — these are things you can Google, not worth taking up memory space.

My retro agent has three hard entry rules, all must be met to store memory:

Non-Googleable: Can't be found by searching online. Excludes all general knowledge.
Codebase-Specific: Can point to specific files, error messages, project-unique patterns. Excludes generic experience.
Hard-Won: Lessons paid for with real debugging effort. Excludes features completed smoothly.

And total memory files are kept at 5-8 — there's a hard cap. New experience coming in means old experience gets merged or eliminated. Prevents memory from bloating into an archive no one reads.

The effect of these three rules: every piece of memory stored is bought with real effort. Next time review retrieves memory for pre-commit predictions, what's retrieved is high-signal information, not noise.

Decision Five: Cross-Vendor Design Review — Catch Blind Spots Claude Can't See

Although review and qa run in independent subagents, they run on the same model family. This means they share certain common biases from training data — preferences for certain design patterns, insensitivity to certain error types.

Hence third_party_review: before writing any code, use another vendor's model to independently review design.md.

Implementation is via headless Claude Code + switching ANTHROPIC_BASE_URL to a third-party endpoint — DeepSeek, Kimi, or any service compatible with the Anthropic Messages API works. Configure a provider.env file (containing endpoint + key + model), the script automatically launches an independent session, feeds in design.md, requirements.md, and the project's CLAUDE.md, and writes the complete review report to disk.

Note this agent's design positioning: advisory, does not block /ship. It's not a gate — it gives you another perspective. A design that seems "obviously correct" to Claude might have three failure modes you didn't consider from DeepSeek's view. After reading the report, you can choose to go back to /architect to fix the design, or accept the risk and continue /ship.

Section 3: The Orchestration Layer — Making the Review Gate a Real Gate

The four designs above are about individual agent decisions. But the most critical engineering decision in the entire system isn't in any single agent — it's in the /ship orchestrator.

/ship isn't an agent; it's a state machine. What it does looks simple: chain dev → review → qa into a loop, exit when green. But the implementation details determine whether this system is a "real gate" or "going through the motions":

First, review and qa run in subagents, not the current session.

Dev executes in the current session — you can watch in real-time what code it changes, interrupt and correct at any time. But review and qa use the Task tool to open independent subagents with isolated context windows. This means the review agent only sees design.md and implementation.md — it can't see what you changed midway and then reverted, nor the dev agent's "inner monologue." It can only judge based on documents.

This point is subtle but important: if review and dev shared the same session, review would be "contaminated" by dev's thought process — it would unconsciously follow dev's narrative. Independent context forces review to maintain irrelevance.

Second, the review gate actually blocks.

Most CI/CD so-called "gates" are purely advisory — if it fails, you manually override and move on. The gate here is a hard block:

After review returns, the orchestrator extracts Critical and Important blocking counts
Any item > 0 → skip this round's QA, directly send back to dev
Dev must dispose of all blocking items (fix Critical, fix or whitelist-defer Important) before entering the next round

There's an easily overlooked design here: skipping QA when blocked saves tokens. Running QA with known defects is meaningless — QA will only repeatedly discover the same issues. Let dev clean up first, then let QA verify.

Third, the loop has a hard cap protection.

At the end of round 4, pause and ask the user — "4 rounds have run, X issues remain unresolved, continue?" Exceeding 5 rounds forces termination, listing remaining issues + root cause analysis (design flaw / implementation capability insufficient / requirements themselves contradictory). This doesn't represent distrust of the agent. The real purpose is to prevent the system from burning tokens infinitely in a state of "almost done but always one step short."

Fourth, documents are the interface protocol between agents.

All agent outputs strictly land in the <feature-name>/ directory, forming the "seven-piece set": requirements → design → (third_party_review) → implementation → review → test_report → progress → retro. review.md and test_report.md use incremental append — new chapters insert at the top of the file in multi-round loops, historical content pushes down. Modifying/deleting historical chapters is prohibited.

This means you can trace the entire decision evolution of a feature: what loop 1 review found → how dev disposed of it → what loop 2 review verified → which issues were ultimately accepted, which were fixed. This documentation isn't for archiving. It's starting context for the next feature.

Section 4: The Real Bottlenecks of This System

At this point, you might think "this system looks pretty complete."

No. Here's its most honest defect list right now:

First, when loops pile up, the context window starts getting tight. Every agent needs to read design.md, implementation.md, the previous round's review.md, and test_report.md. Two rounds are manageable; three or more rounds, just reading historical documents costs tens of thousands of tokens. For review.md, we've done incremental append: new chapters insert at the top, history pushes down, old chapters untouched. This preserves traceability, but the cost is files growing longer and longer. I'm currently experimenting with automatic summary compression of historical chapters, but it's not live yet.

Second, review and qa sometimes collude. Theoretically they're independent agents, but in practice they both run on the same model family, with some blind spots identical. That's why third_party_review exists — switching to a different vendor's model for design review, specifically catching Claude's shared biases. But this only reaches the design layer; cross-model review at the code layer hasn't been done yet.

There's another more practical problem: this process is too heavy for simple tasks. Fixing a one-line typo doesn't need to go through clarify → architect → dev → review → qa. I've consciously made a judgment: small tasks don't run the full pipeline. What kind of task counts as "small"? Currently, it relies on intuition. This is the next problem to engineer.

Fourth, the retro-to-memory closed loop isn't tight enough yet. Theoretically, memory should be referenced at the design stage of the next feature — architect reads memory to avoid repeating mistakes. But currently, the integration between architect and memory is still relatively weak, relying more on the review stage's pre-commit predictions to utilize memory.

Finally, and most fundamentally — this system assumes you can write a good CLAUDE.md. If the project's CLAUDE.md is blank or outdated, all agents work in the wrong context. This isn't a loop problem, but it's the prerequisite for the loop to be effective — garbage in, garbage out.

Section 5: What Really Matters Isn't the Code Loop, It's the Evolution Loop

If you remember only one thing, remember this:

The most important loop in this system isn't the dev → review → qa code loop. It's the retro → memory → next feature evolution loop.

The code loop ensures this time no mistakes. The evolution loop ensures next time is stronger than this time — and it's automatic.

Six months ago, when I first pieced this thing together, it was an error-prevention pipeline — confirmation bias, review fatigue, memory evaporation, the three most common weaknesses of solo development, each assigned an agent to watch. At that time, memory was empty, and review's pre-commit predictions relied entirely on "intuitively feeling where errors are likely" — hit rate about half.

Six months later, memory holds seven hard-won incident patterns. Not nonsense like "write tests" — but "in this project, the time-alignment logic for PIT financial data drifts when crossing markets," "the type of error last review missed was async race conditions, because the design didn't include a sequence diagram." Now when review starts, it first retrieves relevant patterns from memory, then lists predictions. Hit rate went from fifty percent to over seventy percent. This doesn't rely on model upgrades — Opus is still the same Opus. It relies on:

The system remembers where it fell last time, and looks in that direction first this time.

That's why memory entry rules are so strict — not because storage is expensive, but because noise poisons this evolution loop. Store 100 pieces of "watch out for boundary conditions" level platitudes, and the next retrieval is noise, hit rate actually drops.

Few but high-quality memories are the fuel for evolution; many but messy memories are the resistance to evolution.

This is also why the "pre-commit prediction" design is the soul of the entire system — not because it catches many bugs. It's because it creates a measurable calibration loop. Every time review ends, you can compare how many predictions hit, how many were missed. The types of problems missed are the system's cognitive blind spots. And the moment a cognitive blind spot is exposed is the entry point for the next evolution.

Writing this far, back to the opening question — what's the fundamental difference between Loop Engineering and Prompt Engineering?

The difference isn't whether there's a loop; it's whether the loop itself learns.

A dead loop without memory, without calibration, without self-improvement is just a script configured with retry logic. It will repeat the same mistake N times, consume N times the tokens, and then coincidentally pass on a random round. This kind of loop has no accumulation — every time starts from zero.

A loop that learns accumulates across three dimensions with every iteration:

Knowledge accumulation (memory): what type of defect you tripped on last time, store it, prioritize checking next time.
Calibration accumulation (prediction hit rate): missed problem types expose cognitive blind spots; blind spots become priority attention areas for the next round.
Process accumulation (iteration of agent definitions): you found clarify dumping eight questions at once produced poor quality, so you added the "one question at a time" constraint. You found Minor suggestions always got skipped, so you added the "low-risk fix" rule. Every agent's current definition is the accumulated design judgment from dozens of previous loop iterations.

This is what makes Loop Engineering truly valuable. It's not saving labor — it's turning every development experience, including failed experiences, into permanent system upgrades. Six months ago it was an error-prevention pipeline; now it evolves on its own.

What it will look like a year from now, I don't know. But I know it will definitely be stronger than now — because every feature run, it grows a little bit across three dimensions. This isn't a one-time efficiency boost; this is a compound interest structure.

Closing — Open Source Not Because It's Perfect, But Because It's Not

The entire system is open source: github.com/Peakstone-Labs/claude-ship

git clone https://github.com/Peakstone-Labs/claude-ship.git
cd claude-ship
./install.sh   # copies into ~/.claude, immediately usable

After that, in any project: /clarify <feature> → /architect <feature> → (optional /third_party_review <feature>) → /ship <feature> → /retro <feature>.

If you want to enable cross-vendor design review, copy third-party-review.d/provider.env.example to <provider>.env, fill in the third-party endpoint information.

Open source isn't because I think it's perfect. It's precisely because it's not — the five defects listed in Section 4, every one exposed through real usage. The purpose of open source isn't "publishing best practices"; it's letting more people use it, then telling me where the design is wrong. Same as my quant system — build in public.

If you're already using a similar workflow, or after reading this article want to try, a few quick-start suggestions:

Start with review and retro, not all seven at once. These two agents have the highest ROI — review prevents defects from entering the codebase, retro turns every fix into the system's permanent memory. Other agents (clarify, architect, qa, third_party_review) add when you feel "yeah, I really need this now."
Write a good CLAUDE.md first. All agents' effectiveness depends on the accuracy of project context. Spending an hour writing a good CLAUDE.md is worth more than spending a day tuning agent prompts.
Treat the first feature's memory as investment, not cost. After the first feature runs, retro might store nothing — because the three entry rules are strict. Normal. By the second, third feature, memory slowly accumulates. The first three features' loops look "low cost-performance," but they're tilling the soil for the fourth and beyond.

This system's design goal was never "perfection" — Section 1 said it, Section 4 listed five unsolved defects. The design goal is today a bit stronger than yesterday, tomorrow's feature steps on one fewer landmine than today's. Six months in, it's delivered.