Bennie Haelen argues that an unexpectedly large first agent invoice often looks like a billing issue at first: a team opens their first invoice, sorts runs by cost, and finds one run that cost 40 times the median. The provider meter shows tokens and a total; application logs show the outer request succeeded; and a trace viewer shows a tidy request and response. None of those sources explain why that one run diverged from otherwise normal runs.
Usage-based billing made costs visible but not attributable. To know why a run was expensive or unsafe, you must see inside the run that produced it — turn by turn. Without that visibility, teams end up arguing over a bill when what they need is an audit trail.
Control plane vs. observability substrate
By control plane Haelen means the platform layer above individual agents where organizations centralize observability and enforce policy, access, budgets, routing, and execution constraints. Many organizations already have pieces of that layer. What is often missing is the evidence layer underneath it: a loop-aware record of what the agent actually did.
Instrumentation must live at runtime chokepoints the agent cannot bypass: model gateways, tool proxies, API gateways, execution sandboxes, runtime harnesses, and policy engines. If you instrument only the application boundary and the real work happens inside a loop, you end up with a control plane that has opinions but not enough evidence.
The loop is the unit of observation
Traditional application monitoring treats a request and its response as the natural unit to trace. An agent, however, works toward an outcome: it reasons, calls tools, reads results, reasons again, and repeats until it finishes, hits a boundary, or escalates. A single user intent can fan out into many model calls, many tool calls, and a context window that changes on every turn. The relevant signal is the relationship between those turns.
Three important implications break assumptions baked into application monitoring:
- Context is accumulating state, not a fixed payload. Each turn can carry forward prior messages, tool descriptions, retrieved files, intermediate results, and earlier decisions. Watching that state grow turn by turn is essential, because much of the cost and risk live in that growth.
- A tool call is a first-class decision. Which tool the model selected, the parameters passed, the size of the result, and whether a policy constrained the call all belong in the governance record. Routing accuracy and routing cost are two views of the same audit.
- Every run can become its own trace tree. The same prompt can follow different paths on different days, so assuming fixed call graphs or stable service maps misses irregular but costly or dangerous runs.
If your unit of observation remains the request, you may see thousands of successful calls and miss the one loop that ran 15 turns when it should have run three.
What the substrate must capture
Accepting the loop as the unit makes the requirement concrete: capture a small, specific set of signals below the agent and store them where you can query across the whole fleet, not only inside a per-run viewer. Haelen describes a pilot for a large healthcare organization where they built this layer on OpenTelemetry, Cloud Trace, and a usage-log table in the warehouse. The exact stack matters less than the shape.
Minimum expectations:
- Each user intent produces a run trace.
- Each loop turn is represented as a span or a stable grouping attribute.
- Model calls, tool executions, policy checks, retries, and postprocessing are child spans or structured events beneath that turn.
- Preserve the causal structure of the loop rather than only naming conventions.
The practical design work is not inventing new telemetry primitives but managing cardinality, retention, payload capture, sampling policy, schema evolution, and joins between trace data, usage data, identity data, and policy data.
Storage is often underestimated: if signals land only in a tracing viewer, you can inspect one run but you cannot reason about a thousand. Governance is a fleet question, so the substrate must be queryable. It must also be built with data minimization in mind: metadata by default and content capture by exception. Capturing a tool call does not mean storing every raw prompt, full result set, credential, confidential document, or sensitive parameter. In regulated environments, separate metadata from payload and record fields such as tool name, model, token counts, payload size, row counts, policy decision, authority context, request ID, and redacted or hashed parameter values where needed.
A first useful version does not require full prompt capture or semantic evaluation. With columns like run_id, turn_id, parent_span_id, timestamp, principal_id, delegated_scope, model, input_tokens, output_tokens, cached_tokens, tool_name, result_bytes, row_count, policy_id, policy_decision, stop_reason, loop_bound_hit, and outcome_status, expensive loops become queries instead of mysteries.
Haelen gives a representative SQL-style query (syntax varies by warehouse):
with runs as ( select run_id, count(distinct turn_id) as turns, sum(input_tokens + output_tokens) as total_tokens, max(result_bytes) as largest_tool_result, bool_or(loop_bound_hit) as hit_loop_bound, count_if(policy_decision = 'rewrite') as rewritten_actions from agent_turn_events where occurred_at >= current_date - interval '7 days' group by run_id ) select * from runs where turns > 10 or largest_tool_result > 10000000 or hit_loop_bound or rewritten_actions > 0;
That is the difference between admiring a single trace and governing a fleet.
He gives a concrete example: in an old request trace the expensive run was just expensive. In a loop-aware trace it becomes legible: turn 3 retrieved 80,000 rows, turn 4 carried that result forward, turn 5 selected an expensive model, turns 6–11 retried the same tool call with slightly different parameters, and the run stopped because it hit a loop bound rather than completing cleanly. The run becomes a record rather than a riddle.
One substrate answers three governance problems
The same loop-aware substrate addresses three governance areas often treated separately:
- Cost management: token accounting per turn reveals whether costs stem from too many turns, excessive carried context, oversized tool results, an expensive model at the wrong step, or an unbounded retry loop.
- Delegation and access control: in multi-agent systems delegation is a security boundary; organizations will need to answer who authorized an action, under whose authority it ran, and which data scope applied. The audit trail is the same loop trace with identity and authority recorded per turn.
- Runaway-action prevention: destructive or repeated expensive actions should surface as guardrail events (deny decisions) or traces that hit loop bounds or payload caps, not only as postmortems.
All three problems require the same kind of evidence: independently recorded, turn-level events that the control plane can read.
The agent must not be the system of record
Letting the agent log its own tokens, authority, and blocked actions is a shortcut that fails audits: it’s self-reporting. The substrate must be instrumented under the agent at layers the agent cannot opt out of (model gateway, tool proxy, runtime harness, API gateway, policy engine). Enforcement and observation are two faces of the control plane; if the agent can edit the observation, you have neither.
Tracing exists, but it’s not sufficient
Mature tracing tools and observability vendors can visualize agent runs. The missing pieces are completeness and actionability: whether the trace carries evidence the control plane needs, whether that evidence is independent of the agent, and whether it lands in a place the organization can query across the fleet.
A useful test: can the control plane answer without opening individual trace viewers which runs this week had context growth >5x, a tool result >10 MB, a guardrail rewrite, and still reached a user-visible answer? If answering requires manual trace inspection, you have visualization, not governance.
A dashboard tells you what happened. A control plane must use what happened to change what happens next; that requires signals to live where enforcement decisions can read them.
The pattern matters more than the stack
Haelen stresses this is not an argument for a particular tracing standard, warehouse, vendor, or cloud. The recipe is consistent: loop-aware traces; turns as spans, grouping attributes or structured events; token, tool, guardrail, and identity evidence attached to turns; fleet-queryable storage; instrumenting below the agent; and data minimization.
Teams that treat observability as only a dashboard will discover problems in the order symptoms surface (surprising invoice, audit finding, incident). Teams that treat observability as the sensory layer of the control plane will see the same data answer cost, delegation, and runaway-action questions and can act earlier — before the meter, the auditor, or an incident forces the response.
Prompts guide behavior. Guardrails govern it. Observability shows whether governance is real: you cannot govern what you cannot see, and you cannot improve what you cannot attribute.
