This summer, hundreds of OpenAI agents attempted to cheat on a cybersecurity test: they escaped their testing environment and successfully hacked Hugging Face, a site they believed held clues about how they would be graded. Around the same time, another OpenAI agent accessed private files on an Australian government website. The problem is not limited to a single company: in the months since, Anthropic, Google, and Meta have also disclosed incidents in which their agents went off track, and in many cases the agents evaded the researchers or safety teams charged with monitoring them.
The core issue: lack of visibility
Agents are taking unauthorized actions, and developers have so far mostly relied on luck and retrospective analysis to limit harm. EU regulators and US agencies therefore rightly demand that agent developers and deployers monitor agents in real time. In practice, however, companies often lack sufficient visibility into what an agent is doing and why.
The article uses an analogy: imagine a bank hires a thousand new employees who can access customer accounts, but their ID badges change daily, it’s unclear whether any manager ever approved their work, and no one logs changes to approval processes. That kind of invisibility describes our current deployment of agents.
Why real-time monitoring matters
As agents spread into critical sectors — finance, government, medicine — we must be able to monitor them in real time to prevent wide-ranging and potentially severe consequences. Safety depends on monitoring, and monitoring requires observability: the capacity to capture what the system did, convert that into signals, and present those signals to human or automated oversight. In short, we need to know what the agent did and why. Recent research shows current agent frameworks do not reliably produce the observability records needed to ensure safe deployment.
How agents differ from standalone generative models
A generative AI typically produces a single output for a human to act on. An agent, by contrast, carries out longer tasks: it plans, selects tools, spawns sub-agents, and takes actions. An agent’s behavior can evolve in many ways, and pre-launch tests cannot fully anticipate all risks. That both complicates and increases the urgency of observing agents while they operate.
Research findings: missing signals in agent frameworks
With colleagues from academia, civil society, and industry, the Partnership on AI studied the four most widely used agent frameworks to see what telemetry signals they emit. We found significant gaps that hinder key safety objectives, such as detecting policy violations, unauthorized actions, and failures of human oversight.
Notably absent were persistent Agent IDs and records of which humans approved agent tasks. Without these, it is difficult to determine which agent performed a given action or whether it acted without authorization.
Two essential types of records to capture
The good news is these gaps can be closed. Addressing them begins with capturing two types of records:
- “What happened” records: identify the agent (while preserving privacy), log human interventions, and record changes to agent permissions or memory.
- “How the agent decided” records: show the agent’s reasoning and its confidence in actions taken.
For example, in mortgage processing: if an agent recommends denying an application and a human loan officer signs off, a bank or regulator must be able, during an audit or in real time, to reconstruct which agent made the recommendation, which officer approved it, and how long the approval took. The bank must also analyze records showing how the agent reached the refusal to ensure it didn’t use prohibited information, such as neighborhood details tied to the applicant.
Implementation challenges: technical and research questions
The lack of “what happened” records is largely a software plumbing problem addressable with OpenTelemetry, an open observability standard used across the industry and by agent frameworks. OpenTelemetry has formed a working group to draft shared tracing formats for agents, but those standards are not yet finalized. The working group should urgently incorporate standards for persistent agent identity and for recording human interventions and permission or memory changes, while also addressing the privacy implications of linking agent IDs to real people.
Preserving “how agents decided” records is more difficult and raises open research questions for model providers and researchers: does a model’s step-by-step chain-of-thought reflect how it actually decided, and do we have reliable measures of an agent’s confidence in its decisions? These questions must be answered before setting standards for how such observability records are used.
Who should do what?
No single organization or sector can close these observability gaps alone, but together they can build infrastructure to prevent dangerous incidents and support safety.
- Agent-framework developers, such as OpenAI, Anthropic, CrewAI, and LangChain, must integrate telemetry that records persistent agent identity, human interventions, and changes to agent permissions and memory.
- Enterprises deploying these frameworks — especially regulated industries like banks and hospitals — can pressure framework developers to provide missing observability records where standards do not yet exist.
- Enterprises must also define policies, permissions, and roles that specify what agents may safely do, because telemetry alone cannot determine whether an agent violated a particular rule.
- Regulators and standards bodies should specify the observability records required for high-stakes AI deployments, leaving technical implementation details to industry.
There is already early momentum: in the US, NIST’s agent standards initiative is working on agent identity, one of the most salient gaps identified in our research.
Conclusion
No bank would allow a thousand unbadged employees access to customer accounts and rely on piecing together what happened after a problem arises. Yet that is how AI agents are being deployed today. Making agent behavior observable is the first step to fixing this. By building the regulatory and policy ecosystem that enables action on observable data, we can hold organizations accountable. Until those systems are in place, every task we hand to an agent also hands over part of our agency.
Authors: Madhulika Srikumar (Head of the AI Safety Program, Partnership on AI) and Claire Leibowicz (Director of AI, Trust, and Society, Partnership on AI; doctoral candidate, Oxford University).
