Specs Alone Won't Save You: Building a Harness That Keeps AI Agents Honest

If someone had asked me half a year ago how to get AI to maintain a project long-term, I probably would have answered: write clear requirements, write a PRD and technical plan first, don't rely on just a single prompt.

Now my answer has grown by half: Specs are important, but specs alone are far from enough.

I used Claude Code to maintain a self-hosted family asset management system. From the repository snapshot of May 6, 2026 to August 2, 2026, it accumulated a total of 536 commits. The tech stack is Java 21, Spring Boot 3.3, MyBatis, MySQL, Thymeleaf, and HTMX; currently, the production Java code is about 30,700 lines, and the repository contains 24 PRDs, 25 technical designs, 52 database migrations, and 438 JUnit tests.

There was already plenty of documentation and quite a few tests. But the project still encountered the following situations:

• The PRD clearly stated "switching display currency must not change the rate of return," yet the same type of bug was fixed three times;
• A guard test showed "broker entry exists," but the user said the feature was gone;
• The MySQL health check was green, but the application kept restarting due to a wrong password;
• The backup script printed "success," but when it was time to actually restore, it was discovered that the so-called .sql.gz had never been gzipped at all.

They share one commonality: The implementation passed a certain criterion, but that criterion was not verifying the result the user actually needed.

This is also the reason I later began to seriously understand Harness Engineering. The problem is not just "did the AI understand the Spec," but whether the entire repository provides the Agent with a reliable working loop: where to get context, how to operate the environment, what evidence to use to judge results, how to correct after failure, and where to stop when reaching the production boundary.

Prompt, Spec, and Harness are not three synonyms

I now distinguish them like this:

Prompt  = What to do this time
Spec    = What result counts as correct, and explicitly what not to do
Harness = The entire engineering environment that the Agent can read, operate, verify, and correct to complete a task

Expressed more concretely:

Harness
  = Navigable repository knowledge
  + Reproducible runtime environment
  + Tools the Agent can use directly
  + Layered criteria close to real results
  + Feedback that allows iteration after failure
  + Permission gates that cannot be crossed

Spec is an important input to the Harness, but it is not the whole thing.

You can write "must be usable on mobile" a hundred times in a PRD. If the Agent cannot see the mobile layout, cannot start the application, cannot take screenshots, and has no criteria to check for occlusion and horizontal overflow, this sentence still relies on a human to finally discover the problem.

You can also write "backups must be reliable." If the acceptance check only looks for a .sql.gz file in the directory, the Agent can easily get an all-green but unrecoverable system.

In the Harness engineering article published by OpenAI in 2026, the engineer's work is summarized as designing environments, expressing intent, and establishing feedback loops that allow Agents to work reliably; one key judgment is: from the Agent's perspective, things that cannot be read and verified at runtime effectively do not exist.

Anthropic, in Demystifying evals for AI agents, also distinguishes two concepts: the Agent harness allows the model to receive input, call tools, and execute tasks; the evaluation harness is responsible for running tasks, recording processes, judging results, and summarizing evaluations.

When it comes to ordinary business projects, I don't think it's necessary to get overly hung up on terminology boundaries. The real question that needs answering is: After an Agent completes a change, on what basis does it know it did it right?

The four "false greens" I stepped on were all cases of choosing the wrong Oracle

The testing field often calls the "mechanism for judging whether a result is correct" a Test Oracle. AI Coding amplifies this problem because the Agent will very actively optimize towards the criteria you give it.

If the criterion only checks for a string, it will make the string exist; if the criterion only checks for a file, it will make the file generate; if the criterion only asks if a service responds, it will make the service respond.

As for whether the user can find the entry, whether the file can be restored, whether the credentials can actually log in—these are not within this criterion.

1. The Spec was written correctly, but local tests did not cover system invariants

This system supports three display currencies: CNY, USD, HKD. The rules are not complex:

Amount type: amount(view) = amount(base) × fx
Ratio type: ratio(view)  = ratio(base)

After switching net worth to USD display, the values should scale, but the rate of return, debt ratio, asset allocation percentage, and emergency reserve months must not change.

This rule was always written in the documentation, and the AI could explain it correctly in every conversation. But the actual code was still fixed three times: sometimes only the numerator was converted, sometimes the multi-period window lacked an anchor rate, sometimes the third currency lacked triangular conversion via the base currency.

The problem wasn't that the Spec wasn't clear enough, but that verification always stared at a few fixed values on the current page. Later, I changed the criteria to properties: for the same business fact, generate three currency views; amounts must scale by the factor, and all ratios must be exactly equal. Below is the core assertion simplified from the real repository test CurrencyInvarianceTest, omitting test data construction and other amount fields:

@Test
void ratiosStayInvariantWhileAmountsScale() {
    KpiSnapshot cny = kpisFor(BigDecimal.ONE, "CNY");
    KpiSnapshot usd = kpisFor(new BigDecimal("6.774"), "USD");

    assertThat(usd.debtToAssetRatio())
            .isEqualByComparingTo(cny.debtToAssetRatio());
    assertThat(usd.monthlyInvestReturnPct())
            .isEqualByComparingTo(cny.monthlyInvestReturnPct());

    assertThat(usd.netWorth())
            .isEqualByComparingTo(cny.netWorth()
                    .multiply(new BigDecimal("6.774")));
}

The change in this layer of Harness is: from "remember this rule" to "any new metric must pass through the same invariant." The AI doesn't need to re-understand a discussion from three years ago; if it breaks something, it will receive definite failure feedback.

2. A link existing in the DOM does not mean the user can see it

The project once had a static guard called "broker entry on account page":

grep -q '/broker(id=' accounts/index.html

Later, a UI cleanup tucked the "Broker" button into a ⋯ menu with no text. The link was still in the template, so the guard always passed; the user directly asked me: "Is this feature gone?"

To the user, "still there but can't find it" and "already deleted" are no different.

The Oracle needed here is not string existence, but after real rendering on both endpoints, the entry is visible, not occluded, and doesn't require expanding a hidden container first.

Now the project uses scripts/entry-points.json to register capability entries and runs browser checks in two layouts: PC 1440×900 and mobile 390×844:

a.scrollIntoView({ block: 'center', inline: 'nearest' });

const rect = a.getBoundingClientRect();
const hasArea = rect.width > 1 && rect.height > 1;
const collapsed = !!a.closest('details:not([open])');
const inMoreMenu = !!a.closest('.row-more-pop');

let top = document.elementFromPoint(
  Math.round(rect.left + rect.width / 2),
  Math.round(rect.top + rect.height / 2)
);
while (top && top !== a) top = top.parentElement;

const visible = hasArea && top === a && !collapsed && !inMoreMenu;

The original grep was not deleted; it is still suitable for quickly discovering if a link was accidentally deleted. It just no longer impersonates a user visibility test.

The change in this layer of Harness is: Static facts and runtime facts are verified separately, and test names cannot promise things beyond their own capability.

3. The database responding does not mean the application can use the database

In the Docker installation process, I once used mysqladmin ping to check if the database was ready. Once, after a user re-cloned the project, the random password in the new .env did not match the password in the old data volume, causing the application to constantly report Access denied.

Strangely, the database container always showed Healthy, and the entry script also printed "MySQL ready."

After reproducing it, I discovered that mysqladmin ping can still return exit code 0 when the password is wrong. The question it answers is "is the server responding," not "can this set of credentials complete a database operation." The same error primitive was reused by the Compose healthcheck and the entry script, so both layers were false green together.

Old criterion:

mysqladmin ping -h db -u "$DB_USER" -p"$DB_PASS"

New criterion:

mysql -h db -u "$DB_USER" -p"$DB_PASS" \
  -Nse 'SELECT 1'

After a real query fails, the script won't continue guessing "the database might still be initializing," but will identify the authentication error and enter a credential synchronization process without deleting business data; if it cannot be safely repaired, it stops.

There is another easily missed Harness design point here: After changing a criterion from lenient to strict, check who consumes it as control flow. The first time I changed the healthcheck to a real query, depends_on: service_healthy caused docker compose up -d to exit non-zero, and set -e terminated the script before the self-healing logic. The Oracle was corrected, but the call chain didn't keep up.

So a criterion upgrade must check at least three things: whether the proposition it judges is correct, whether the failure information is sufficient for repair, and how upstream and downstream will consume this failure.

4. A backup file existing does not mean the backup can be restored

Another time was even more dangerous. The backup script generated a file named finance-xxx.sql.gz, du -h could see its size, so the script printed "backup successful."

When actually performing the restore, this appeared:

gzip: not in gzip format

The reason was simple: the Docker branch wrote the mysqldump output directly into .sql.gz; the filename had .gz, but there was no gzip in the middle at all. Worse, the before-restore-* fallback automatically created before restoration also reused the same erroneous logic.

The original Oracle was: file exists and size is greater than 0.

Now it at least checks:

mysqldump ... | gzip -9 > "$backup"
gunzip -t "$backup"
gunzip -c "$backup" | grep -q 'CREATE TABLE'

But this still only proves the compressed package can be unpacked and looks like SQL inside. The real acceptance is a closed-loop round trip:

Write marker data
  -> Backup
  -> Delete marker
  -> Restore
  -> Marker must return
  -> Service healthy

The fallback generated before restoration must also be restored once more, proving it can truly undo this restoration.

The change in this layer of Harness is: For low-frequency, high-risk paths like backup, migration, release, and rollback, you cannot just verify that the action occurred; you must verify the promised result.

After four incidents, I finally saw what a Harness should look like

Entry tests, database probes, and backup scripts seem like three unrelated problems. Putting them together, a more stable working loop can be abstracted:

Give the Agent a repository map
  -> Let it complete changes in a reproducible environment
  -> Provide layered feedback using Oracles of different costs
  -> Feed the failure reason and repair direction back to the Agent
  -> Until the criterion closest to the user's result passes
  -> Wait for human authorization before external state changes

Layer 1: Knowledge must be navigable, not all stuffed into context

The project now uses AGENTS.md to record product boundaries, environment topology, page maps, development processes, and cross-module linkages. For example, when changing income sources, the fact layer, family cash flow, Dashboard empty state, and reports must be synchronized; when adding a new ratio metric, it must enter the currency invariance test.

But I also stepped on the downside of the "big comprehensive manual": the current AGENTS.md is already 222 lines, and the early description of "no holdings" once conflicted with capabilities that were later launched. The existence of a document does not mean it is still trustworthy.

A more reasonable structure should be: a short entry serves only as a map; domain knowledge is dispersed into documents that have owners, can be cross-linked, and can be checked for freshness. The Agent first sees the route, then reads layer by layer according to the task, rather than swallowing an encyclopedia at the start.

The next step the project needs to supplement is not more rules, but document structure checks, stale rule scanning, and regular cleanup. This point is also consistent with the experience in the OpenAI Harness Engineering article: "Give the Agent a map, not a thousand-page manual."

Layer 2: Turn the application itself into an object the Agent can read

If the Agent only looks at code, many problems will never be visible. Therefore, the project gives it a work surface that is not just Maven:

• Can start an independent beta, using synthetic data to complete login and user paths;
• Can access database truth values, not just read Controller return values;
• Can read logs, health status, versions, and migration results;
• Can take screenshots in PC and mobile viewports, checking Canvas, occlusion, horizontal overflow, and privacy mode;
• Can run backup, restore, deployment, and diagnostic scripts, and verify side effects.

This is not about adding a few fancy tools for AI, but about reducing the human effort of ferrying evidence in the middle. If an error only exists in a phone screenshot I see, and the Agent cannot see it, that screenshot effectively does not exist for the current execution process.

Layer 3: Establish feedback with a cost gradient

Not every change needs to run a full end-to-end test right from the start. The current project roughly has this feedback gradient:

Compilation / Type checking
  -> JUnit formulas and domain invariants
  -> Static guards (cross-file linkage, forbidden items, migration structure)
  -> HTTP + database truth end-to-end mainline
  -> Browser dual-endpoint rendering and screenshots
  -> Beta user paths
  -> Release, image, health check, and rollback verification

The earlier it is, the faster it is and the more direct the location; the later it is, the closer to the user's result, but the higher the cost.

The goal of the Harness is not to wrap everything in the heaviest tests, but to make errors fail as early as possible at the cheapest and sufficiently realistic layer. What static grep can judge doesn't need a browser; but user visibility absolutely cannot stop at grep.

Layer 4: Failure information itself is also context for the Agent

In traditional CI, a single test failed might be enough for a human to trace the stack. When an Agent is working, the error information should try to directly tell it: which invariant was broken, what the evidence is, and which files to look at.

For example, the entry check won't just print FAIL, but will distinguish: the link is simply not in the DOM, the element has no area, it is occluded by another element, it is hidden in the ⋯ menu, or the secondary page entry path does not exist.

A good Harness doesn't just close the door; it also puts the information needed for the next round of correction back into the loop. Otherwise, the Agent will just guess again.

Layer 5: Passing verification does not equal obtaining release permission

The project allows the Agent to autonomously modify code, execute tests, and commit, but tag, push, and production release are another type of external state change.

The release process first executes a preflight, checking workspace, version, migrations, tests, and documentation linkage, and then must stop. Only when the maintainer returns a confirmation string exactly matching the target version:

release vX.Y.Z

does the process continue to tag, push, backup, migrate, deploy, and verify. If the version does not match or there is no reply, it cannot proceed.

This gate is not responsible for judging code quality; it is responsible for expressing the boundary of responsibility: The Agent can prove "I think it can be released," but cannot deduce from that "I have obtained authorization to modify the production environment."

How to truly sediment an incident into the Harness

I no longer just add a "be careful next time" after every error. A more reusable processing sequence is:

1. Identify the old criterion that passed this time but actually lied
2. Clearly write the result the user actually depends on
3. Choose the observable evidence closest to that result at the current cost
4. First put the fault back, proving the new criterion will turn red
5. After fixing, prove it will turn green
6. Write the failure reason and repair entry into the output
7. Retrospective audit: how many of the existing guards are still making the same type of mistake

Step 4 is very important. Running a green light only on the fixed code cannot prove that the test can really catch the problem.

Step 7 is also very important. The project had already recorded in v1.6.14 that "display does not equal visible," but at the time only treated it as a reminder for writing new tests in the future, without going back to check the existing entry guards. As a result, by v1.6.23, the same type of error appeared again on a different page.

A lesson being written down does not mean the lesson has taken effect. Only when an old error reappearing can automatically fail does it truly enter the Harness.

This Harness is still not good enough

I can't pretend the problem is solved by writing this far.

First, repository knowledge is still too centralized. AGENTS.md is too long, and drift can still occur between PRDs, technical designs, READMEs, and page engineering numbers. Currently, the test counts in different documents have already gone out of sync, which shows that the Harness for "check document numbers before release" has not yet covered all authoritative sources.

Second, there are many static guards, and some of them are still based on grep. They are cheap and effective, but also easily misled by comments, example text, and identical strings. High-risk criteria must be continuously upgraded to the semantic, database, or runtime layer.

Third, browser checks return SKIP when there is no Chromium or the application is not started. A development machine is allowed to honestly SKIP, but if formal CI also accepts SKIP, this line of defense is just decoration.

Fourth, real broker synchronization requires the user's own account and credentials, which cannot be fully reproduced in public CI. Currently, only mocks can cover the reconcile logic, and the maintainer does real-machine confirmation in beta. This is a clear verification gap that cannot be masked by the number of unit tests.

Harness Engineering is not "set up a test platform once," but continuously searching for places in the system that still require human brain supplementation, human ferrying, or produce false greens, and then gradually turning them into capabilities that the Agent can read, execute, and fail.

Finally

Spec Coding solves: don't let AI write code directly when requirements and boundaries are unclear.

Harness Engineering continues to ask: even if the requirements are clear, how does the Agent obtain the correct context in a long-term project, how does it verify real results, how does it continue to correct from failure, and how does it avoid crossing the boundary of human authorization.

Model capabilities will continue to strengthen, but the faster the model can generate code, the more important the quality of the feedback loop becomes. Because it not only amplifies correct implementation, but also amplifies wrong Oracles, stale documentation, and false green lights.

The cases in this article come from the open-source project "Family Account House" that I maintain. It is a self-hosted family asset snapshot, return attribution, and risk analysis tool. The following Harness artifacts are all in the public repository:

• Project map and invariants: https://github.com/LuoDi-Nate/financial-management/blob/master/AGENTS.md
• Feature entry runtime criteria: https://github.com/LuoDi-Nate/financial-management/blob/master/docs/entry-points.md
• QA cases and incident records: https://github.com/LuoDi-Nate/financial-management/blob/master/docs/qa-cases.md
• Deployment, backup, and restore scripts: https://github.com/LuoDi-Nate/financial-management/tree/master/deploy
• Complete source code: https://github.com/LuoDi-Nate/financial-management

Conflict of interest statement: I am the project maintainer. The project uses Apache 2.0 and has no paid version. The project data in the article is based on master@67deb53 from 2026-08-02; public pages and screenshots are all synthetic demo data, not real family assets.