Agentic systems often trade accuracy against cost: the highest-performing proprietary frontier models are expensive, while smaller or more efficient open models start with lower accuracy. Fine-tuning addresses this but requires expertise and infrastructure. An alternative is formalizing and verifying harness-level tuning—using evaluation benchmarks designed for a given harness and per-model harness profiles such as LangChain’s agent harness profiles.
Goal and prerequisites
This tutorial shows how to create and validate a LangChain Deep Agents harness profile for NVIDIA Nemotron 3 Ultra that makes the model behave closer to higher-performing proprietary models on specific agent workflows. All adjustments are made at the harness level using existing Nemotron 3 Ultra endpoints provided by NVIDIA cloud providers.
Prerequisites:
- A host with Python and LangChain Deep Agents installed.
- An API key for NVIDIA Nemotron 3 Ultra (build.nvidia.com offers free API access for testing; production endpoints are available on Baseten, Crusoe, Fireworks, Nebius, Together AI, etc.).
- Recommended: a LangSmith account for collecting agent traces.
Manual tuning and verification loop
LangChain Deep Agents provides two key tools for model-specific harness tuning: an open-source evaluation benchmark and agent harness profiles as a first-class extension point. The manual tuning procedure is:
- Establish a baseline by running the benchmark with a deep agent and Nemotron 3 Ultra without a harness profile.
- Analyze failures.
- Propose harness profile changes that address the failures.
- Re-run the benchmark to verify that changes improve performance without regressions.
Profile levers include:
- Prompts: change base system prompts, prompt suffixes, or tool descriptions (for example, instruct Nemotron to prefer clarification questions or to prefer tool results over model recall).
- Exclusions: remove tools or middleware.
- Additions: add middleware or sub-agents (for example, middleware that checks for truncated model responses or incorrect tool names).
The engineering goal is to make agent-to-model calls resemble the model’s training inputs more closely.
Example failure: read_file pagination
NVIDIA Nemotron 3 Ultra failed a test: tests/evals/test_file_operations.py::test_read_file_truncation_recovery_with_pagination[nvidiahub:ultra-tme]. The expected final text contained 'opal-fox-91', but the model returned only 'x'. The first read_file call returned the first page, which contained only 'x' values; the expected answer appeared later. The model should have continued reading with offset/limit pagination but answered from the first page.
The read_file tool is central to coding, data analysis, and other agentic tasks in LangChain Deep Agents. The diagnosis: the model treated the first page’s last line as EOF because the page did not signal that the file continued.
Proposed fix (middleware summary)
The suggested fix was a ReadFileContinuationNoticeMiddleware that annotates read_file results when the returned page reaches the per-read limit. The middleware:
- Only acts on read_file tool calls.
- Counts returned lines and compares to the limit.
- If lines >= limit, appends a notice to the model-visible content indicating the file likely continues and instructs to call read_file again with offset=offset+limit.
The middleware was added to the harness profile’s extra_middleware list and the profile registered with register_harness_profile("nvidia:nemotron-ultra", profile).
Validation and results
After applying the middleware, the failing test and the full benchmark were re-run. Reported results:
- Baseline: Read file tests 0 / 3, Average 94 / 127
- With read file middleware: Read file tests 3 / 3, Average 96 / 127
The middleware resolved all three failing read_file tests and improved the overall benchmark score from 94 to 96 out of 127.
Overfitting considerations
Before finalizing changes, overfitting must be considered. The pagination middleware is a mechanical fix that teaches the agent to make appropriate tool calls and is less likely to overfit. By contrast, adding domain-specific definitions to fix a semantic confusion (e.g., "incident alerts" vs "incident triage") would more likely cause overfitting.
Automating harness profile creation
The manual loop—run benchmark, inspect failures, change profile, re-run—is well-suited to automation. Geoffrey Huntley calls a similar pattern the "ralph loop": a model repeatedly reads state, makes one change, and uses the filesystem rather than conversation as memory. The same loop applies to harness engineering: propose a change, check it against an objective signal (the evaluation benchmark), keep or roll back, and repeat.
LangSmith Engine is LangChain’s managed implementation of this loop: it detects failures in production traces, diagnoses roots in source code, drafts fixes, and adds evaluators to prevent regressions. Every proposed change is human-reviewed before shipping.
The article provides a small reference implementation (NemoClaw community repository) that tunes a single harness profile against an evaluation benchmark. Key reliability decisions for the improvement loop are:
- Verify before believing: require a fix to pass the same test several times to filter out lucky passes.
- Snapshot and roll back: save the profile before attempts and restore if unsuccessful.
- Learn from the last try: feed the previous attempt and its failure back to the proposer so the agent refines rather than repeats mistakes.
- Re-check the whole suite: after any accepted fix, re-run the entire benchmark to detect regressions and new failure modes.
The loop continues until the profile passes cleanly, a round produces no improvement, or the iteration budget is exhausted.
The agentic proposer
In the "propose a change" step, a model-driven proposer receives a description of the failure, tools to explore solutions, and writes a bounded profile modification. Each proposal runs in a short agent session where the agent has:
- A writable copy of the profile (only this file is editable).
- Read-only access to the failing test, observed agent trajectory, SDK, and benchmark code.
- File tools (read, edit, grep, glob) to confirm behavior before changing the profile.
The agent makes the smallest change that addresses the diagnosed failure and writes a short explanation. Two constraints keep proposals grounded: write access is scoped to the profile file, and the agent is instructed to prefer general fixes over test-specific hacks. Additionally, parts of the evaluation suite may be held out as a validation set the proposer never sees to reduce overfitting.
A sample run log shows the reference implementation using a ralph-style loop with Nemotron 3 Ultra: baseline 20 passed, 1 failed (correctness 0.95); the agent diagnoses the pagination root cause, proposes middleware and prompt changes, verifies 3/3 runs, marks FIXED, and after re-running the full suite reports 21 passed, 0 failed—i.e., 20 → 21 passed and 1 → 0 failed.
Adapting to other harnesses
This automation pattern is not LangChain-specific. To adapt Nemotron Ultra to other agent frameworks, expose three programmatic interfaces: (1) run evaluations and report per-test pass/fail and trajectories, (2) provide an editable and programmatically validatable profile/config file, and (3) use a frontier-quality model to propose edits. With these, the propose-verify-keep/rollback-re-run loop operates unchanged.
Further resources
- The Nemotron 3 Ultra agent harness profile published by the LangChain team.
- The NVIDIA NemoClaw Blueprint for LangChain Deep Agents, which includes Nemotron and NVIDIA OpenShell.
- Nemotron 3 Ultra access on Baseten, Crusoe Cloud, DeepInfra, Fireworks, Nebius and Together AI.
- Documentation on how NVIDIA optimized a LangChain Deep Agent for the Deep Research Bench by tuning harness, model, and workload.
Closing
Harness engineering combined with evaluation benchmarks and per-model profile extension points enables measurable, automatable improvements to agent behavior without retraining models. The approach is auditable (snapshots and rollbacks), verifiable (repeat verification), and adaptable across frameworks that expose evaluations, editable profiles, and a proposer model.
