On September 3rd, the ARC Prize (a comprehensive exam testing reasoning abilities approaching human limits) announced the latest results of ARC-AGI-3, with data that defies intuition.

The same model—GPT-6 Astra—operating at identical reasoning intensity, delivered vastly different results under two different Harness setups:

In the standard Harness, it scored 62.7%; switched to the Provider Adapter Harness, it scored 98.6%.

Note that this is not a comparison of two models but two tests of the same model, with a 35.9 percentage point difference.

More counterintuitive is the cost. The higher-scoring setup incurred lower expenses: dropping from approximately $26,100 to $17,300, a reduction of about 34%.

Without changing the 'brain,' merely altering the work approach nearly transformed the Agent's performance level.

Days later, YC hosted a dedicated Paper Club event titled 'Why the Harness Matters More Than the Model.' Moderated by YC's Franois Chaubard, the session featured researchers from three distinct perspectives: Seth Karten from Prime Intellect, Jon Saad-Falcon from Stanford, and YC's own Josh France and Regan Bell.

While their research focuses varied, all converged on the same subject: the Harness—the framework surrounding the model.

The basic conclusion: The key to winning in AI competition is shifting from 'whose model is stronger' to 'who can better organize the model.'

Model capability is becoming a 'given condition.' Under the same condition, Agents can still differ by 36 percentage points. This gap represents the value space of the Harness.

So, what exactly is a Harness? And why can it create such a significant difference? This article unpacks the insights from the discussion.

/ 01 / Harness: The 'Work Approach' Surrounding the Model

The term 'Harness' originally means 'horse tack'—the apparatus that channels a horse's strength to a carriage.

In the AI context, it refers to the entire operational framework enabling a large model to complete real-world tasks: how system prompts are written, how context is organized, which tools can be invoked, where memories are stored, how sub-Agents are divided, and how conversations are managed.

A vivid distinction: the model determines 'how smart the brain is,' while the Harness determines 'what tools the brain has at its disposal.'

When an Agent executes a task, which materials it can access, which tools it can use, and whether it can continue from a previous step are all determined by the Harness, not the model.

YC Partner Franois Chaubard experienced this transformation firsthand.

In March, while experimenting with Karpathy's auto-research project, he initially aimed to add an interface to clarify what the Agent was doing and the experiment's progress. Gradually, material retrieval, experimentation, review, writing, and progress management were integrated—unintentionally creating a complete Harness.

Now, he only needs to provide research directions and evaluation metrics; subsequent work is continuously advanced by multiple Agents. Papers produced in March and April were relatively rough; later, he could input multiple research ideas simultaneously, let the system run, and return later to collect results of considerable quality.

The model remained unchanged, but the Agent's delivered results improved steadily.

This reflects a simple truth: the model sets the upper limit, while the Harness determines whether that limit can be reached.

A note on context: Harness-related work has long been considered 'unglamorous' in the machine learning community. Adjusting prompts, connecting tools, and structuring task loops sounded more like engineering optimization than model research, with some even openly questioning whether it qualified as genuine AI research.

Yet now, this overlooked aspect directly impacts performance.

/ 02 / The Boundary Between Harness and Model Is Being Bridged

In recent years, Harness development focused on adding capabilities. Now, the Harness itself has become an optimization target.

First, prompts were optimized. Previously, poor prompts required manual trial and error. Now, systems like DSPy can experiment with different wordings and select the most effective version based on test results—transforming 'prompt tuning' from a craft into a search problem.

Second, the Harness itself was optimized.

Darwin Gdel Machine (DGM) went further: it not only tunes prompts but can directly modify the Harness code running the Agent. If a task workflow underperforms, it rewrites it and evaluates the new version based on test results. In paper experiments, this system raised SWE-bench scores from 20% to 50%.

Third, historical experience was incorporated into the next iteration.

Continual Harness added another layer: Agents can review past task records and results to decide whether to modify prompts, add skills, update memories, or adjust sub-Agent configurations.

In other words, experience from previous tasks now informs the next Harness version.

A more radical current direction involves enabling the model itself to continue learning during operation—retraining the Agent on newly generated data and even updating model weights directly during testing.

What does this mean?

Optimization is shifting from the Harness toward the model. Agent progress now depends on Harness iteration.

In other words, the boundary between Harness and model is being bridged.

/ 03 / A Good Harness Doesn’t Need Rigid Processes

A counterintuitive finding: as models grow stronger, the Harness doesn’t need to predefine every step.

Traditionally, Agent design involved breaking tasks into fine-grained steps: what to do first, which tool to invoke second, how to evaluate results third. While this approach offers stability, it falters when tasks lengthen or environments change, as predefined processes easily become obsolete.

Seth from Prime Intellect introduced the team's Prime Agent in his talk. Its approach is the opposite: minimize predefined processes, maximize resources available to the model.

A core design is 'information stratification.' Agent-needed information is stored in three distinct places.

The most critical, currently used information is placed directly in the context; if historical information is too lengthy, it’s compressed to retain only what’s truly needed for the current task.

Programs, computation results, and task progress are saved in a persistent REPL—a programming environment that doesn’t clear after a single invocation. Whatever code the Agent wrote, results it calculated, or task progress it made can be directly resumed later.

Long-term memories, skills, and prompts are stored externally, not constantly loaded into the context, and retrieved only when needed.

Sub-Agents follow the same design. After completing a task, they aren’t immediately 'cleared.' Their accumulated context can be retained, allowing them to resume previous work when reactivated.

In essence, Prime redefines responsibilities between Harness and model. This design shines in long-duration tasks.

Seth’s team had an Agent play Factorio continuously for seven days—a highly complex automation management game requiring players to plan production lines, allocate resources, unlock technologies, and adapt to potential errors that could alter subsequent routes.

Over seven days, the Agent invoked 633 sub-Agents, generated over 23 million output tokens, completed 24 of 196 technologies, and advanced 'advanced circuits' research to 71%.

A severe mistake once caused progress to regress from 5 completed technologies to just 1. However, the system didn’t clear its state and restart.

Previous programs, resource statuses, and task records remained intact, allowing the model to reassess the situation and continue progressing.

This highlights the Harness’s core value in long-duration tasks. Such tasks encounter numerous unforeseeable situations, requiring the model to decide the next step based on current results, while the Harness preserves prior states and work outcomes to enable continuous decision-making.

However, granting the model greater autonomy doesn’t mean the Harness can replace the model itself.

On the ARC-AGI-3 public set, Prime paired with Claude Opus 5 achieved a RHAE score of 95.5%; switched to Terra, it scored only 25.7%.

The same Harness with different underlying models still produced a nearly 70 percentage point gap.

Thus, the accurate statement is: the Harness is crucial, but it cannot replace underlying model capability. It amplifies existing model abilities rather than creating them from scratch.

/ 04 / Weak Models Rely on Harness for Support

A more practical scenario is localizing personal AI.

Today, many personal Agents depend on cloud-based large models. While they handle writing, research, programming, and scheduling, the cost is direct—long-term API fees can reach thousands of dollars, and personal data like emails and files must be continuously sent to the cloud.

Stanford researcher Jon Saad-Falcon introduced OpenJarvis in his talk to address this: bringing personal AI back to local devices as much as possible.

The challenge is that local models aren’t yet strong enough to directly replace cloud models.

Saad-Falcon noted that local models lag behind cutting-edge models by roughly 6–12 months in capability. Moreover, this gap isn’t closed by simply replacing Claude with an open-source model.

They conducted a straightforward experiment: replacing Claude Opus 4.6, originally used by OpenClaw and Hermes Agent, with Qwen3.5-9B while keeping everything else unchanged. Accuracy dropped by 24.8 and 38.8 percentage points in two tests.

So, can part of this gap be closed from outside the model?

OpenJarvis’s approach is to optimize the entire AI system. It divides personal AI into five layers: model selection, model execution, Agent operation, tool and memory access, and ongoing system learning.

Each layer can be adjusted independently and recombined. For example, a stronger cloud model can first analyze failure cases to identify issues and help the local system modify configurations; after optimization, the local model handles daily operations.

In short: the large model judges, the small model executes.

How effective is this? With Qwen3.5-9B entirely unchanged, the optimized system recovered approximately 56% and 77% of performance losses in the two tests.

Across eight more comprehensive tests, the best local solution averaged 80.3% accuracy, compared to Claude Opus 4.6’s 83.5%—a gap narrowed to 3.2 percentage points. In four tests, the local solution matched or surpassed the best cloud results.

Critically, this was achieved at minimal operational cost. By these tests’ metrics, the local solution’s marginal API cost was roughly one-eight-hundredth of the cloud’s, and end-to-end latency was reduced to about one-fourth.

OpenJarvis proves that personal AI’s shift from cloud to local doesn’t require small models to fully catch up to large ones; the Harness itself can first close part of the capability gap.

/ 05 / When Agents Enter Companies, Harness Manages Operations and Permissions

The final scenario is enterprises.

When Agents truly enter companies, the question isn’t just 'can they work?' With dozens of Agents running simultaneously, how they’re deployed, maintained, and what information they can access or modify becomes equally critical.

YC’s internal Agent exploration began in early 2025. Initially limited to system prompts, tools, and task loops, it later integrated Slack, scheduled tasks, and virtual machines, enabling Agents to modify code, run tests, and even possess 'personal computer'-like work environments.

However, when YC scaled this capability company-wide, problems emerged.

The team once deployed over 50 Hermes Agents in virtual machines, each requiring individual configuration; when an Agent failed, engineers had to log into each instance to troubleshoot and repair. As Agent numbers grew, maintenance costs soared.

YC’s Josh France and Regan Bell introduced their internally developed Harness—QM—to address such scalability issues.

QM’s core change: Agents no longer 'reside' in a single machine. Conversations, context, and long-term states are centrally stored, while virtual machines and isolated environments become resources invoked as needed. Agents can select different machines based on tasks or even switch model providers.

At this stage, the Harness’s role expands beyond 'connecting tools for a single Agent' to managing state, resources, and operation modes for a group of Agents.

However, after scaling, real office environments expose three unavoidable issues.

First, the Agent may mistake a local issue for a global one.

YC once tried to have the Agent identify bugs and modify the system on its own based on operational records, with mixed results. The team referred to one failure mode as the "protagonist syndrome": the Agent only saw the small problem it encountered but might propose modifications that affected the entire system. Currently, such modifications still require human oversight.

Second, the Agent gives up too easily.

Despite having time and tools available, it may declare a task a failure after just a few attempts. YC addressed this by creating a Grind Tool—setting a minimum runtime or Token budget for tasks, preventing the Agent from ending before the threshold is reached.

Third, and most troublesome, is permissions.

Once the Agent enters Slack, it can see a vast amount of context but may not truly understand which information can be shared with whom. Humans know not to casually forward private conversations between colleagues to another group, but without clear permission boundaries, sensitive information may flow between different conversations for the Agent.

YC’s current approach is to keep most database permissions read-only. When write access is needed, the Agent first proposes a plan, which is then confirmed by a human.

However, human review is not a panacea. As employees grow to trust the Agent more, they may become accustomed to clicking through approvals—turning the review process into a mere formality.

This means that what enterprise Agents truly need to develop in the next phase may not be more capabilities but a Harness that can both scale the Agent's operations and effectively manage permissions.

Over the past two years, the industry's focus has been almost entirely on models. This is not wrong—models remain the foundation. However, moving forward, differences will increasingly emerge above that foundation.

Models determine how fast you can go, but the Harness determines whether you can keep going and how far.

Written by / Yuanyuan