Agent Output Quality Depends on the Harness, Not Just the Model

Recently, I've been using Agents for more and more things.

Writing code, organizing materials, creating event plans. Sometimes I hand off a task, come back a while later, and the result is already usable. Other times, I watch it busy itself for a long time, invoke quite a few tools, the process looks lively, but the final deliverable is completely not what I wanted.

At first, I would attribute this gap to the model not being stable enough.

Later, after using them more, I felt that wasn't entirely it.

With the same model, whether the task is clearly stated, how the context is prepared, how tools and permissions are configured, and who checks the results all directly affect the final outcome.

Very often, what truly creates the gap is not switching to an even stronger model, but whether a proper working environment has been set up for the Agent — that is, the Harness.

On Wednesday evening, leveraging the third session of AI Quan Wen, I combined questions from community members and went through these issues in a focused way.

This session discussed five questions: how to define a task, prepare context, divide the human-machine boundary, and how to solidify an effective collaboration.

The Same Agent, Why Do Results Vary So Much Between Users?

Some people use an Agent, and it generates a lot of content with neat formatting, but after reading it, it feels somewhat hollow. To get the desired result, they have to constantly supplement and correct, and each use feels like starting over.

Others use it very smoothly. The Agent knows what they want, understands their habits, and the results it gives get closer and closer to real work needs.

The gap doesn't necessarily all come from the model.

If you just send an Agent a single sentence and wait for the result, you are essentially still using the Agent as a more complex large model.

Whether an Agent can truly work depends on things beyond the model.

How the task is defined, how context is provided, whether tools can be invoked, at which nodes humans participate, how results are verified, what to do after a failure — all of these together constitute the Harness.

So I very much agree with a formulation proposed earlier:

Agent = Large Model + Harness.

The large model determines the upper limit of capability; the Harness determines whether these capabilities can be stably brought to bear in real tasks.

How Does a Vague Idea Become an Executable Task?

We often find it easy to say to an Agent:

Help me run this account.

Help me analyze the market.

Help me organize customer information.

These sound like tasks, but they are really just ideas.

When a person receives such a request, they might interpret it based on experience or ask follow-up questions. But an Agent easily guesses on its own, and after guessing wrong, it will proceed earnestly with the wrong assumption.

I usually prefer to clarify a task from five aspects:

Background, Objective, Result, Boundary, and Acceptance.

For example, for "help me organize customer information," you at least need to supplement: what this information will be used for, which communication records to organize it from, what type of customers to identify, what format the final output should be in, and what content must not be inferred independently.

If every judgment must cite the original text as evidence, and unconfirmable content is marked as "pending human judgment," the Agent is less likely to fabricate details out of thin air.

This set of methods isn't actually new.

In the past, project management and product design also required tasks to be clearly communicated. 5W2H and SMART were originally meant to solve these kinds of problems.

AI hasn't made task definition disappear; instead, it has made it even more important.

What Context Should You Give an Agent?

When many people think of context, their first reaction is to provide more materials.

But honestly, more context is not always better.

Dumping dozens of documents in at once doesn't necessarily help the Agent understand more deeply; it might just fail to find the key points.

I prefer to think of context as the Agent's working environment.

It needs to know enterprise materials, product information, and historical cases, but also what specifically needs to be done this time, what the current progress is, which rules cannot be violated, and what kind of result qualifies as acceptable.

This includes both long-term maintained knowledge bases, rules, and personal preferences, as well as the input materials and execution status for the current task.

Especially for long tasks, it's best to "persist to disk" midway through execution.

Record what has been completed, what conclusions have been formed, and what problems remain. Later, even if the conversation gets long or the Agent goes off track, you can come back to check the current state without having to guess everything from scratch again.

So, context management is ultimately not about "giving it all the materials."

It's about letting the Agent see the right information at the right time.

How Should Humans and Agents Divide the Work?

In the past, when discussing AI replacement, it was often divided by job role.

Will front-end developers be replaced? Will product managers be replaced? Will salespeople be replaced?

Later, I felt this unit of analysis was still too large.

I heard a case study at the World Artificial Intelligence Conference. A team used AI to transform an industrial process. They didn't directly do the complete "phone screen protector application," but continued to break it down further, only solving a very small process step like "peeling the film off."

This approach gave me considerable inspiration.

Human-Agent collaboration should also not be divided by job role, but broken down into specific task units.

Within a role, some tasks have clear inputs, decomposable processes, and results that can be verified at a glance, and can be recovered after errors — these are very suitable to hand over to an Agent.

Other tasks involve goal selection, rule conflicts, external commitments, or irreversible operations, and should be controlled by humans.

I usually look at two dimensions: Is the task clear? Is the risk controllable?

The clearer the task and the lower the risk, the more it can be autonomously executed by the Agent. The fuzzier the task and the higher the risk, the deeper the human involvement should be.

Especially for operations like deleting databases, making payments, or publishing externally, it's better not to give too much freedom at this stage.

The time saved might not be enough to fix the resulting mess.

How to Solidify a Successful Collaboration?

This might be the most easily underestimated step.

Many people, after achieving a good result, copy the prompt, put it in a document, and reuse it next time.

It's useful, but often not enough.

Because a task's success might not just be due to a well-written prompt. It might also have used specific materials, rules, tools, human-machine collaboration flows, and acceptance methods. It might also have stepped into a few pitfalls along the way, only to be corrected at the end.

So I prefer to solidify a "Task Capability Pack."

Inside, there shouldn't just be prompts, but also the task definition, required context, execution flow, evaluation rules, and discovered anomalies with their handling experience.

The form isn't that important.

It can be made into a Skill, an Agent role, a workflow, or even a well-organized document.

The key is whether, the next time a similar task is encountered, it can reproduce this success relatively completely.

The process I commonly use is: first review why this time was effective, then remove information specific only to the current client or activity, abstract it into a reusable structure, then solidify it, retest with a different but similar task, and finally continue iterating.

I sometimes get lazy with the retesting step myself.

But just like code you write yourself often fails to catch your own bugs, a method that only works well on the original task might just be "overfitting."

Conclusion

Alright, this review of the AI Quan Wen session is finally published at a normal time. I hope those who didn't catch the live stream can also gain something from it.

Good night, everyone~