From Human Gates to AI Referees: The Third Iteration of AI Coding Workflow Design
AI coding workflows that shift judgment from human confirmation to an auditable, single-turn Referee let teams run more tasks without constant human babysitting, while keeping authority over rules and red cards in human hands. The thin-shell/core split also solves the problem of maintaining identical workflow rules across Claude Code, Codex, and DSH without drift.
The third installment in this series refines three existing AI coding workflows—serial dev-workflow, parallel dev-workflow-router, and bugfix-workflow—by adding an independent AI Referee agent. The Referee is tool-free, single-turn, receives fixed input, and outputs structured JSON, acting as a judgment layer separate from the main Agent and Reviewer. Green and yellow judgments let the workflow proceed without stopping; only red cards and standard changes require human intervention. This directly attacks the bottleneck where every Gate waited for a human.
The Referee's rule evolution shifts from AI self-review to recording human signals: when a similar human correction appears twice or more, the Referee generates a rule-change proposal that a human must approve, and it includes rollback options. Bugfix workflows get a four-axis traffic-light system (cause, solution, evidence, scope) with mandatory checks, where yellow can be explicitly authorized but red cannot. To manage multi-platform use across DSH, Claude Code, and Codex, the design separates a platform-neutral core layer from a thin shell of platform facts, and adds a drift-check script to enforce the boundary. Cost stays predictable: a typical task makes 0-2 Referee calls, maximum 4.
The Referee's isolation is as much a cost-control measure as a correctness measure: keeping it tool-free and single-turn makes each judgment predictable in tokens and recomputable, which is what lets the workflow run unattended.
Making human signals the trigger for rule changes avoids the feedback loop where an AI system rationalizes its own mistakes during retrospective; a human's explicit correction is treated as ground truth, not something the AI re-evaluates.
The thin-shell/core pattern applies software-engineering discipline to AI prompt workflows, acknowledging that AI tools like Claude Code and Codex are deployment targets that need a single source of truth to avoid config drift.
The design assumes most engineering decisions do not require deep human judgment; it works well for tasks with clear acceptance criteria, but the author's own caveat about targeting real engineering suggests the bottleneck shifts to how well humans define the axes in the first place.
'Gate becoming formalized' is a sharp observation. Our trade-off is that daily tasks don't have human Gates; we only involve people in two places: when the agent is uncertain it stops and asks the person who assigned the task, and at the end, reviewing the diff of this branch against the task description; tasks interrupted for more than an hour ask the person whether to continue first, not letting it keep running on stale context. I want to ask: when the Referee judges red and returns to a human, does the human see the original payload or a summary reorganized by the main Agent?
First, to answer the question: The Referee receives the main Agent's payload, which mainly includes a comparison of old vs. new requirements and a comparison of technical solutions. The Referee uses scripts based on accumulated rules to judge each relationship, then aggregates the judgment results back to the main Agent. So note that judging red/yellow/green is the main Agent's job, not the Referee's. After the judgment, the main Agent shows the user a summary. Regarding the Gate settings: 1. After developing with AI for a long time, because humans gradually participate less, many requirements and details are actually forgotten. If a wrong original requirement combined with a wrong new requirement gets developed in, it's a serious contamination of the project, sometimes even harmful. That's why I set up the traffic-light rules. 2. The gate after code is written is because developers need real-device testing. Writing code and running it on a real device are two completely different things. This step can catch the AI's implementation issues from the start. Thanks for your comment!