A research team at Peking University led by Haoran Ye explored how much of an agent harness’s behavior can be distilled into a model. Their Harness-Zero work applied harness distillation: they first developed a specialized harness for a small model (Qwen3.5-9B), used a stronger model to review and minimally correct the small model’s proposed steps, fine-tuned the small model on those corrected runs, and then removed the harness to see what the distilled model could do on its own.
Experimental setup
- The team created an "evolved harness" for the small model. An automated loop (Kimi K3) detected repeated failures on training tasks and added corrective components such as custom tools, middleware rules to block risky actions, skills files, and memory.
- They generated training data by running the small model inside a minimal harness while GPT-5.6 Sol reviewed each proposed step. Sol used a copy of the evolved harness to judge proposals, passing through good steps unchanged and making the smallest possible correction when the model’s step disagreed with guidance.
- The corrected runs were used to fine-tune the small model, producing a distilled model; the evolved harness and the reviewer were then removed for evaluation.
Results
When tested head-to-head on spreadsheet editing, multi-app tool use, and chemistry tasks, the distilled model improved average task success by 21 points over the original untrained small model. That improvement exceeded the gain the original model achieved when paired with the full evolved harness. However, the distilled model did not always outperform the harnessed version: in chemistry tasks requiring candidate generation and molecule validation, the harnessed model led by eight points.
What transfers into the model and what doesn’t
- The distilled model still ran inside a minimal loop with a Bash tool; the authors state their method narrows what the harness must provide rather than eliminating it entirely.
- Deep domain knowledge, candidate-generation logic, and robust context management were harder to transfer. Stepwise procedures (inspect-before-edit, verify-before-complete) are easier to teach than in-depth subject-matter expertise.
Why training on ‘corrected runs’ mattered
The paper demonstrated that the training signal matters:
- Fine-tuning on the stronger model’s own successful runs left the small model’s score unchanged — knowing someone else is better wasn’t enough.
- Fine-tuning on runs the small model completed inside the evolved harness likewise didn’t help — it only shows the model that someone else performs better.
- Giving the reviewer the correct answers produced near-perfect training runs but led to overfitting and little improvement. The effective approach was training on corrected runs: examples that looked a lot like what the small model would have done but with small, achievable fixes. Those corrections were within the small model’s capacity to learn and raised task success by 21 points.
Why models are absorbing harness parts (and what Sutton predicted)
Rich Sutton’s 2019 essay The Bitter Lesson observes a long-term pattern in AI: hand-built problem-specific knowledge helps briefly but is eventually outpaced by general learning methods that scale with computation. Harnesses are hand-built knowledge; model training is the learning process that can absorb it. Sutton also recommends building meta-methods that can discover arbitrary complexity, not embedding solved bits of knowledge directly — for harnesses, that implies capturing and automating the process used to build them.
Multiple industry voices have predicted the same practical outcome: models will internalize many scaffolding behaviors. Examples cited include Jesus Rodriguez’s TheSequence piece "The Great Absorption," Boris Cherny (creator of Claude Code) noting scaffolding gains may be erased by next-model improvements, and Vivek Trivedi at LangChain predicting planning, self-verification and long-run coherence will migrate into models.
What should remain in the harness
Certain responsibilities still belong to harnesses because they tie to tools, data, users and the environment:
- Tool interfaces and validation logic
- Company- or user-specific data handling
- Long-lived context management across requests Industry evidence supports this mixed picture: ARC-AGI-3 benchmarks show major differences in performance depending on harness; Anthropic’s Opus 5.5 prompt guidance suggests removing some scaffolding that models now handle, while adding harness elements like system prompts to prevent unattended agents from stopping early and elapsed-time budgets for multi-agent setups.
How to evolve a harness as models improve
If harnesses must change every model generation, automating that evolution is natural, but unconstrained automation can overfit. Google’s RRSI paper found unconstrained automatic harness evolution improved performance on tuned tasks but didn’t generalize; their mitigation is careful bookkeeping: for every candidate change, record the component touched, the intended fix, the diff, changes in score and cost, and whether the change was kept. Components that stop producing gains are removed.
A practical manual maintenance loop looks like this:
- Tag traces with both harness and model versions so you can attribute which change moved results. Reading full traces gives more insight than scores alone.
- Log every harness intervention and what the model did before the harness changed it — this is data many teams discard but is exactly what Harness-Zero used to train.
- When a new model ships, turn off each harness component one at a time and rerun evaluations. Keep components that still improve scores and delete the rest.
- If you use tools like Arize AX, run identical agent evaluations across harness and model versions and compare side-by-side.
Takeaway
Harness-Zero shows models can internalize many harness behaviors when trained on corrected runs that are within their capacity to replicate, and can even outperform the original model-plus-harness in some tasks. At the same time, domain-specific logic, tool validation, and robust context management remain harness responsibilities. Practically, teams should not discard harnesses, but should version, log, and periodically prune them, automating evolution where possible while guarding against overfitting.
