AI Coding Agents Are Fast. The Harness That Controls Them Is What's Missing.

Introduction: Writing Code and Delivering Software Are Two Different Things

Over the past three years, the way R&D teams discuss AI has changed three times.

At first, the question was: "Can it autocomplete code?" Later, it became: "Can it independently fix bugs, write tests, and submit Pull Requests?" By 2026, the question truly worth asking is:

When AI can work continuously, do we have the capability to make it deliver correctly and stably, catch mistakes in time, and not cause trouble in the production environment?

This is the watershed between Agent and Harness.

An Agent can be understood as "an AI that can work." It can read files, modify code, execute commands, run tests, and submit PRs. A Harness is the entire working system surrounding it, including task descriptions, project knowledge, tool permissions, runtime sandboxes, test gates, approval rules, process records, and post-failure improvement mechanisms.

To put it bluntly: An Agent is like a new colleague with strong execution and high speed, but who doesn't understand the company's history and doesn't have to take responsibility for accidents; the Harness is the organizational environment that allows this colleague to work safely.

What is truly scarce is no longer "switching to a smarter model," but turning vague human experience into an R&D system that machines can understand, execute, and verify.

1. Current Situation: AI Has Entered the Main R&D Process, but the Benefits Are Far from Uniform

AI programming is no longer a novelty for a few.

Google's 2025 DORA survey covered nearly 5,000 technical practitioners, about 90% of whom already use AI at work, with a median usage time of about two hours per day; over 80% of respondents felt personal productivity improved, and 59% believed code quality improved. However, the same report also found that while AI positively impacts delivery throughput, it still negatively impacts delivery stability. DORA's conclusion is restrained: AI is more like an amplifier; it amplifies the capabilities of mature teams and the chaos of chaotic teams. (DORA 2025)

Stack Overflow's survey of over 49,000 developers presents a similar contradiction: 84% already use or plan to use AI tools, and about 51% of professional developers use them daily; but 46% distrust the accuracy of results, while only about 33% trust them. Among Agent users, 69% believe personal productivity has improved, but only 17% think team collaboration has improved. (Stack Overflow 2025)

Putting these two figures together is interesting: Individuals feeling faster does not mean the team is actually delivering faster.

Faros AI's telemetry analysis of 1,255 teams and over 10,000 developers found that high AI-usage teams completed 21% more tasks and merged 98% more PRs, but PR review time increased by 91%, average PR size increased by 154%, bugs per developer increased by 9%, and ultimately, no significant overall performance improvement was observed at the company level. (Faros AI 2025)

Their 2026 follow-up report, expanded to 22,000 developers and 4,000 teams, continued to observe larger PRs, slower reviews, and increased rework and incidents. However, this type of data is observational research and can only indicate phenomena occurring simultaneously; it cannot simply prove that AI is the sole cause. (Faros AI 2026)

Therefore, the real state of AI programming is not a binary choice of "effective" or "ineffective," but rather:

The local efficiency of writing code has clearly improved, but requirement clarification, verification, review, integration, and release have not accelerated synchronously. The bottleneck has simply moved backward.

2. Why Do Various Productivity Data Points Contradict Each Other?

In GitHub's early controlled experiments, developers using Copilot to complete a specific JavaScript HTTP service task saw a 55.8% speed increase. This conclusion is true, but it tested small tasks with clear boundaries, built from scratch. (Microsoft Research)

In Accenture's enterprise experiment, developers using Copilot saw an 8.69% increase in PRs per person, a 15% increase in PR merge rate, and an 84% increase in successful builds. However, this study was jointly conducted by GitHub and Accenture, and metrics like PRs and build counts are only proxy indicators of delivered value, not equivalent to customer value. (GitHub and Accenture)

On the other hand, METR in 2025 had 16 experienced open-source developers work on large projects they were familiar with, recording 246 real tasks in total. The result was that using AI made them 19% slower on average. More notably, developers predicted beforehand they would be 24% faster, and afterward still felt they were 20% faster. (METR 2025)

But this cannot be interpreted as a permanent conclusion that "AI makes developers slower." When METR retested 57 developers, 143 repositories, and over 800 tasks in 2026, they found that more and more participants were unwilling to take on tasks that "prohibited AI use," with 30% to 50% actively avoiding tasks that were painful without AI. Multi-agent parallelism also made it difficult to accurately calculate working hours. Therefore, the researchers explicitly stated that the new results suffer from severe selection bias, and the old experiments no longer represent the effects of current tools. (METR 2026)

These studies do not truly conflict. They simply measured different things:

• For new projects with clear requirements and easy acceptance criteria, AI is often fast.
• For familiar, complex legacy systems, the effort to explain context and correct errors may offset the gains.
• Individual coding speed may increase, but team review and release speed may decrease.
• A developer's subjective feeling, code output, and business results are three different sets of metrics.

3. The Core Problem: What R&D Truly Lacks Is Not Agents, but Four Types of Certainty

1. Lack of Certainty on "What to Do"

Many teams still give Agents input that is just a line from a chat log: "Optimize the permission logic a bit," "Make a page like this one," "Add some tests while you're at it."

A human engineer would fill in the missing information from meetings, historical incidents, colleague experience, and business common sense. An Agent will not. It can only package its guesses into code that looks plausible.

Anthropic's 2026 analysis of about 400,000 Claude Code sessions showed that humans bear about 70% of planning decisions on average, while Agents bear about 80% of execution decisions; the stronger the user's domain experience, the more work the Agent can complete per instruction, and the higher the success rate. (Anthropic 2026)

This shows that human value hasn't disappeared; it has just shifted from "how to write" to "why do it, to what extent, and what must not be touched."

2. Lack of Certainty on Project Context

Code is only one part of the system. The content that truly determines how to make changes is often hidden in architectural decisions, interface contracts, online incidents, grayscale rules, chat logs, and the minds of a few senior employees.

Many teams' so-called "giving AI context" is just stuffing more files into the context window. The result is not smarter output, but information overload, conflicting rules, and a mix of outdated documents.

OpenAI's experience from internal Harness experiments is: what you give an Agent should be a "map," not a 1,000-page manual. They use a roughly 100-line AGENTS.md as a directory, linking to versioned documents on architecture, product specifications, security, reliability, and execution plans, while using CI to check if documents are outdated. (OpenAI Harness Engineering)

The key is not how much context there is, but whether it is accurate, discoverable, and traceable.

3. Lack of Certainty on "How to Prove It's Done Right"

The most dangerous output from AI is not an obvious error, but something that "looks fine."

Sonar surveyed over 1,100 professional developers in 2026: 96% do not fully trust AI code, but only 48% always check it before submission; 38% find reviewing AI code more laborious than reviewing a colleague's code, and 61% have encountered code that "looked correct but was actually unreliable." (Sonar 2026)

"Having someone take a look" does not equal reliable human-machine collaboration. A 2026 study based on real GitHub data found that among a sample of 33,596 Agent PRs, 61.38% received no review at all; among reviewed PRs, 58.77% only had other automated Agents participating. (EASE 2026 Study)

Therefore, verification cannot primarily rely on human patience; it must be turned into automatic evidence as much as possible: type checking, unit tests, contract tests, architectural rules, security scans, UI screenshots, performance comparisons, run logs, and rollback-capable deployments.

4. Lack of Certainty on Permission Boundaries

Completion tools only output text, but Agents can read repositories, execute commands, access networks, modify configurations, and even trigger external systems. It is no longer an editor plugin, but an execution entity within the supply chain.

GitHub's security research shows that malicious content can be hidden in Issues, PRs, or web pages, inducing Agents to read local files, leak tokens, or execute commands. GitHub therefore added protections like branch restrictions, network restrictions, manual merging, session auditing, and sensitive information scanning for the Copilot Coding Agent. (GitHub Security)

Permission design must consider the worst-case scenario: not assuming whether the Agent will make a mistake, but assuming it will eventually read a malicious or misleading instruction.

4. Several Common Cognitive Misconceptions

Misconception 1: Buying a license completes the AI transformation. Licenses solve tool availability, not missing knowledge, messy codebases, weak testing, and inefficient approvals.

Misconception 2: High leaderboard scores mean it can handle our production projects. In February 2026, OpenAI stopped using SWE-bench Verified due to testing flaws and training data contamination; in July, they found issues with about 30% of SWE-bench Pro tasks and withdrew previous recommendations. Public leaderboards can show trends but cannot replace testing on internal, real-world tasks. (OpenAI Feb Audit, Jul Audit)

Misconception 3: Writing code faster means products can go online faster. The true delivery cycle also includes requirement waiting, environment preparation, code review, testing, release, and failure recovery. Accelerating only one segment just piles work onto the next station.

Misconception 4: Human in the loop is inherently safe. If thousands of lines of code are generated at once and a tired engineer clicks approve, that person is just a button, not a control mechanism.

Misconception 5: Opening more Agents will linearly increase efficiency. Without task boundaries, file isolation, and unified acceptance criteria, multiple Agents will only produce conflicts, duplicate modifications, and more review work in parallel.

Misconception 6: AI will reduce the need for senior engineers. Reality is more likely the opposite: as implementation costs drop, requirement judgment, system boundary definition, risk identification, and acceptance capability become more valuable. The real danger is reducing junior positions without designing new cultivation paths, ultimately losing the next generation of senior engineers capable of supervising Agents.

5. Real Cases Tell Us: Leading Companies Win on the Foundation, Not Just the Model

OpenAI used three engineers over five months to drive Codex to generate about 1 million lines of code and merge about 1,500 PRs, claiming the time spent was about one-tenth of handwriting. But this project started from an empty repository and specifically built document indexing, structured architecture, isolated workspaces, browser verification, log metrics, automatic review, and continuous cleanup mechanisms. OpenAI itself explicitly warns: without equivalent investment, one cannot assume the results are replicable.

Spotify's case is more representative. In 2026, over 99% of engineers used AI weekly, 94% self-reported productivity improvements, and PR frequency grew by 76%. But before Agents appeared, Spotify had already built Backstage, a unified tech stack, a component catalog, Fleetshift, and automated migration capabilities, cumulatively merging over 2.5 million automated maintenance PRs. Its backend Agent, Honk, runs in an isolated Kubernetes environment, can only call trusted tools, and verifies changes through multi-OS CI. A recent Java migration was completed by one engineer in three days, whereas it previously required hundreds of teams investing weeks or even months. (Spotify Engineering)

Rakuten reported that the average time-to-market cycle for new features dropped from 24 working days to 5 days; a complex refactoring of a large open-source project was completed by an Agent running continuously for 7 hours, with numerical results reaching 99.9% reference accuracy. However, this is a customer case published by Anthropic, a joint vendor-customer statement, suitable for proving "feasible in a specific environment," but not directly extrapolatable to industry average benefits. (Rakuten Case)

The commonality among these cases is not using the same model, but having similar conditions: clear task scope, unified system structure, controlled tools, timely feedback, and verifiable results.

6. Solution: Build a Minimum Viable Harness in 90 Days

Phase 1: First, measure the baseline; don't rush to expand seats. Select 20 to 50 real tasks, categorized by documentation, testing, simple fixes, cross-module changes, and high-risk business logic. Record the time from order acceptance to going live, manual effort, review rounds, rework rate, incident rate, and model cost. Without a baseline, so-called "efficiency gains" can only be based on feelings.

Phase 2: Turn project knowledge into an executable map. Establish a short entry file pointing to architecture descriptions, key business rules, test commands, directory responsibilities, interface contracts, and common faults. Every rule must have an owner and update time. Rules that can be expressed through types, linting, or structural tests should not just be written in documents.

Phase 3: Establish risk classification. Documentation, test supplementation, and mechanical migrations can be automatically executed at low risk; ordinary business logic must be manually reviewed; authentication, payments, database changes, production configurations, and key operations should have restricted permissions and require multi-person approval. The degree of autonomy should be determined by the cost of error, not by model capability.

Phase 4: Put verification into the Agent's work loop. Require it to first state the acceptance method, then modify the code; before each exit, it must run format checks, type checks, and related tests. If tests fail, continue fixing; if verification is impossible, exit explicitly. "Looks correct" is not allowed to replace evidence.

Phase 5: Control change batch size. Limit the number of modules, files, and scale of changes involved in a single task. Large requirements should first be broken down into independently verifiable small PRs. DORA's AI capabilities model similarly lists small batch sizes, strong version control, internal platforms, and accessible internal knowledge as key capabilities. (DORA AI Capabilities Model)

Phase 6: Isolate permissions and runtime environments. Default to using temporary sandboxes, short-lived credentials, network whitelists, and independent branches. Separate read permissions from write permissions; authorize commits, merges, deployments, and database operations separately; all commands, tool calls, input sources, and approval processes must be logged.

Phase 7: Use failures to improve the Harness in return. Don't just re-prompt once. Attribute each repeated error: if documentation is missing, supplement knowledge; if the wrong file is found, supplement the code map; if out-of-bounds modifications occur, add structural rules; if tests miss detections, supplement acceptance cases; if permissions are too broad, tighten tools. LangChain, keeping the same model unchanged, only adjusted prompts, tools, and verification loops to raise the Terminal Bench 2.0 score from 52.8% to 66.5%, demonstrating that the external working system itself is part of capability. (LangChain 2026)

7. What Metrics Should Be Watched?

Don't treat lines of code, call counts, or PR counts as core results. More meaningful metrics are:

• Total cycle time from requirement readiness to production availability;
• The proportion of AI output acceptable without major rework;
• First-time verification pass rate and average correction rounds;
• Manual review time, and the investment of senior engineers within it;
• Rollback, hotfix, and code rewrite rates 7 and 30 days after merging;
• Total model, compute, and labor cost per successfully delivered task;
• Changes in online defects, security issues, and stability;
• Whether the team delivered more real user value, not just more code.

The most important metric can be condensed into one sentence: Whether the human time saved by AI exceeds the verification, rework, and incident costs it creates.

Future Outlook: R&D Competition Will Shift from "Model Race" to "System Race"

On August 19, 2026, OpenAI independently open-sourced Codex's Harness, explicitly treating context management, tool invocation, sandboxing, approval, process state, and cross-turn execution as reusable infrastructure. This is a strong industry signal: the competitive boundary of Agents is shifting outward from the model itself to the complete working system. (OpenAI, 2026-08-19)

Three things are more likely to happen next.

First, models will increasingly become replaceable engines. The assets companies truly accumulate will be internal task sets, acceptance criteria, knowledge maps, tool interfaces, permission systems, and failure data.

Second, code writing will gradually cease to be the main bottleneck. What will be truly scarce are high-quality requirements, user judgment, architectural constraints, verification capabilities, and decision-making speed.

Third, "fully automated R&D" will not become mainstream in the short term; "conditional autonomous execution" will become popular first. Low-risk, verifiable, easily rollbackable work will be highly automated; high-risk work will be researched and implemented by Agents, with humans responsible for goals, trade-offs, and final authorization.

So, moving from Agent to Harness is not just another trendy term. It represents a shift in R&D thinking:

In the past, we strived to make AI write more code; now, we need to build a system that allows it to work only within the correct scope, continuously receive real feedback, and prove results with auditable evidence.

What AI truly lacks when entering the R&D process is not a more diligent "digital programmer," but an engineering control system that can carry speed, constrain risk, and accumulate organizational knowledge. Whoever builds this system first will be able to truly turn model capability into delivery capability.