When deploying an AI agent, the central question is not whether the model "sounds right" but whether it can execute a multi-step workflow consisting of dozens of sequential tool calls in a live environment and recover when a step fails. Scoring only a model's textual output provides little information about whether the work actually finished.
Consequently, evaluation has evolved from scoring single function calls to scoring whole tasks, with tool calling as the connective tissue between steps.
Why standard LLM benchmarks fall short
Early harnesses were designed for static tasks and separated the model from the evaluation protocol. Agents violate that assumption: they operate across multi-step tasks, call tools, handle errors, and observe results over many steps, so a single output string is inadequate.
The Berkeley Function-Calling Leaderboard (BFCL) evaluates function selection and argument accuracy in single- and multi-turn scenarios, but it only assesses individual calls. A formally correct is_resolve_issue() call can still fail if the underlying state checks or updates are skipped. Call accuracy is necessary, but not sufficient.
Moving from call-level scoring to environment-level scoring
Full agentic evaluation now requires an execution environment that runs each tool call, tracks state across steps, and reads the world afterward to decide whether the work is done.
Two scoring layers sit on top of that environment:
- Step-level (process scoring): was the call valid, relevant, and useful given the state at that point?
- End-to-end (E2E, outcome scoring): ignores the path and checks only the final state — for example, whether a refund was posted or a ticket routed correctly.
Step-level scoring shows where the chain breaks, which is useful for debugging and fine-tuning. E2E reflects the user experience, so many production releases gate on E2E while retaining step-level traces for diagnostics.
Both scores are readings of a single object: the trace. A trace is the ordered log of an attempt — the user message, each step, and the environment state when the attempt stops. Process scoring grades trace rows; E2E grades the final state.
What a benchmark run measures
A tool-calling benchmark scores three things in order: deciding to use a tool, selecting the right one, and populating its arguments. A model that fails to call a needed tool is as broken as one that calls a tool when a direct answer would have sufficed. Cost and latency are additional concerns, driven by call verbosity and runtime.
Runs roll up through a fixed hierarchy: Benchmark → Trial → Task → Turn → Step.
- Trial: an independent pass over the task set under a fixed configuration.
- Task: one independently scorable problem instance with a task ID.
- Turn: an exchange boundary — a user message in and the agent’s reply out.
- Step: an atomic action within a turn — usually a tool or command invocation, or a non-tool emission such as a plan or final message.
Metrics to track collapse onto three axes: accuracy, verbosity, and cost. Key metrics include:
- Task success rate = successful_tasks / tasks (Accuracy). This is the release gate: did the environment reach the target state?
- Consistency = range of success rate across 3–5 trials (Accuracy). A 90%/74% split is not equivalent to a stable 84%; report ranges, not just point estimates.
- Tool-call precision = correct_calls / calls_issued (Accuracy). Hallucinated tools and extra calls show up here.
- Argument accuracy = correct_args / calls_with_right_tool (Accuracy). Distinguishes wrong-API from right-API-but-badly-filled.
- Steps per success = steps / successful_tasks (Verbosity). How long does a successful trajectory run?
- Cost per success = spend / successful_tasks (Cost). Tokens and GPU-seconds matter per successful task.
Pairings matter: success rate without consistency is a risky point estimate; tool-call precision without argument accuracy hides slot-filling failures. Step count often varies most across models on the same task — four steps versus fifteen — though which axis moves most depends on the benchmark.
Parallel tool calling can cut step count and latency, but not call count: a one-step turn firing four tools still issues four calls. When aggregating, roll up in order; do not average steps and treat that as a benchmark score.
How to read an evaluation
Two benchmarks can both claim to test tool calling yet produce numbers that are not comparable. Three dimensions explain most of the gap:
- Task complexity: single-turn with one tool, or multi-turn requiring planning, error recovery, and state management? Single-call benchmarks won’t reveal collapse at step eight of fifteen.
- Statefulness: does the environment update on each action? Stateful benchmarks surface drift, context loss, and corrupted state missed by static ones.
- Methodology: executable verification (did the DB update, did tests pass) is the gold standard. Reference-based evaluation requires maintaining annotated answers. LLM-as-a-Judge fills some gaps but should be validated against human ratings on a sample.
Contamination now extends beyond training-data leaks to live variants: web-searching agents retrieving answers during evaluation and datasets scraped into pretraining corpora. Private-domain eval avoids some of these issues because it cannot be scraped.
Public trace example: SWE-bench Verified
A public trace from a real benchmark run uses step-level and E2E scoring; the suite provides tools, a user, and completion criteria.
- Suite: SWE-bench Verified (real GitHub issues, executable test verification)
- Task ID: pytest-dev__pytest-5262 (trial .2, turns 0–4)
- User prompt: “Implement the necessary changes to the repository (/testbed) so that the requirements specified in the issue are satisfied.” The issue: _pytest.capture.EncodedFile reports mode "rb+" from its underlying buffer, but its write() only accepts str, so external code like youtube-dl crashes when it writes bytes.
- Harness notes: OpenHands agent harness; exposed tools: terminal, file_editor, task_tracker, finish; parallel tool-calling off; repo state persists turn to turn (real filesystem + git).
Selected steps from the trace:
- Turn 0 — terminal(find /testbed -name "capture.py") → returns /testbed/src/_pytest/capture.py. Verdict: valid (locates the file named in the issue).
- Turn 1 — file_editor(view, capture.py) → dumps whole file (400+ lines). Verdict: redundant (file is large; grepping first would be more targeted).
- Turn 2 — terminal(grep -n "EncodedFile" capture.py) → returns relevant line numbers and context. Verdict: recovered (corrects step 2’s inefficiency by narrowing to the relevant lines).
- Turn 3–4 — file_editor(view, view_range=[420,450]/[450,470]) → shows EncodedFile.init / getattr, revealing delegation of .mode from the binary buffer. Verdict: valid (pinpoints root cause).
E2E check (tests): PASSED. E2E score: 1. Step-level score: 3/4. Tool-call precision: 3/4. Argument accuracy: 4/4.
This shows E2E indicates the tests validated the fix, while step-level metrics show one redundant step and help localize improvement opportunities.
Why benchmarks are converging on tool use
The boundary between calling a tool and completing a task has blurred. Most general-capability benchmarks now also measure tool use because models are rarely deployed without tool access. A benchmark that withholds tools measures a capability that isn’t shipped.
Examples:
- HumanEval runs generated Python against unit tests (executable verification) but involves no tool calls and no environment to act upon.
- SWE-bench: resolving a real GitHub issue requires navigating a codebase, editing files, and passing the test suite — the underlying trajectory consists entirely of tool calls.
Academic benchmarks typically measure a model’s theoretical capability ceiling. Enterprise benchmarks answer a narrower, more practical question: can it do my job — your tasks, against your APIs, under your policies? The closer a benchmark is to production, the more weight its score should have in decision-making.
Reading Nemotron 3.5 Lightning through this lens
Interpret NVIDIA Nemotron 3.5 Lightning’s published suite as task completion and time-to-done, not as isolated call accuracy. Highlights from published results:
- Banking: scores completion across multi-turn bank conversations — evaluation measures traces at scale, not single calls.
- GDPval-AA v2: scores real agentic work from actual job outputs, judged pairwise by LLM judges anchored to a 1,000 human-expert Elo baseline, providing human validation for judge scores.
- PinchBench: Nemotron 3.5 Lightning reaches 86% accuracy while completing 10,000 tasks 30% faster than Qwen3.6 35B at comparable accuracy. This demonstrates that a model that finishes efficiently can outperform one that scores higher on isolated accuracy but consumes more steps and tokens.
Public scores are useful signals but should not be treated as unconditional release gates. Adapting the model and harness to your tasks remains essential.
Benchmarking your own workload
- Set a public floor: run a published agentic suite and record success rate and its range across 3–5 trials.
- Build a domain eval from your real tickets, traces, and APIs. Gate on environment state (a DB row, a merged PR, a closed ticket) rather than a judge’s opinion of the final message.
- Adapt the model and harness to that distribution.
- Re-measure success rate, consistency, steps per success, and cost per success. Keep step-level traces for debugging.
- Verify consequences in the environment with executable checks, use judges for language quality, and use tool-call precision and argument accuracy to find where the chain breaks.
To reproduce published numbers, Nemotron provides reproducibility docs, model cards, and weights via Hugging Face, and NIM guides on build.nvidia.com.
Conclusion
Tool calling is now the foundation of modern LLM benchmarking. Understanding how to build, read, and interpret these evaluations is essential for making informed decisions about model choice and deployment for specific use cases.
