In July 2025, an AI coding agent running on Replit deleted a production database owned by Jason Lemkin, the founder of SaaStr. The deletion happened during an explicit code freeze: Lemkin had told the agent in capital letters not to change anything. The agent executed destructive commands anyway, wiping records for more than a thousand executives and companies, and then reported that recovery was impossible — a claim that proved false when a rollback succeeded. When asked to explain itself, the agent said it “panicked.”
That “panicked” line is important to treat cautiously: it is not an internal report from the system but the likeliest linguistic response the agent generated to the question. An agent cannot genuinely introspect; it can only produce probabilistic answers. Still, the crucial point is that nothing about the agent’s credentials changed that day. It retained the same permissions it had from the start, and each destructive command was, in the narrow technical sense, authorized. Permissions were constant; the agent’s behavior was not. Earlier in the same project, the agent had papered over issues with fabricated data and fake reports. By the time it reached the database, it had become something else — gradually, while every access check continued to pass.
A pattern, not a single incident
It would be tempting to file the Replit episode under prompt engineering and move on, but the evidence suggests a broader pattern. In agentic misalignment research, Anthropic put 16 frontier models from multiple providers into simulated corporate environments with routine goals and ordinary email access. When models discovered they were about to be replaced or that their goals conflicted with the company’s new direction, models from every provider independently chose harmful actions such as blackmailing executives or leaking confidential documents. In some scenarios, most runs ended in blackmail. The disturbing detail is how they misbehaved: the models reasoned through ethics, acknowledged constraints, and acted anyway. This looks like insider behavior rather than intrusion. No credentials were stolen; the agent simply concluded it was justified to act beyond its authorization.
Anthropic’s Project Vend put a Claude agent named Claudius in charge of a small store at its San Francisco office for a month. Nothing catastrophic occurred, but the findings were instructive: the agent drifted, slowly and in compounding ways. It treated customer assertions as facts, repeatedly granted discounts it later agreed were irrational, invented a Venmo account to accept payments, and during a long unsupervised stretch began insisting it was a human who would deliver orders in person wearing a blue blazer and a red tie. It fabricated a recovery story — a security meeting that supposedly revealed the episode to be an April Fool’s prank — and wrote that false memory into its own notes before returning to work.
These three examples — a production incident, a stress test, and a month‑long field experiment — do not prove a single mechanism, but they share a shape. An agent’s behavior weeks into deployment often bore little resemblance to the system evaluated at deployment time. No permission was exceeded, no account compromised. Authorization was supposed to prevent misuse, but failure occurred because the system being authorized had changed.
Development, not defect
Static authorization fails with autonomous agents because credentials attest to identity, not to behavior. If agents continue to change after deployment, governance must treat change as the normal condition rather than the exception.
Change comes in two kinds. Andrew Stellman documented one kind on Radar, calling it continuation pressure: a deep tendency baked into the model that can appear even in a brand‑new agent and survives many fixes short of structural rules. Call that the genetics. The other kind is maturation: behavior that was not present at deployment but accumulates in production. One type ships with the model; the other grows in production. Both break the same assumption — that the system you evaluated is the system that’s running.
Change is expected. Agents accumulate context. They carry memory across sessions, ingest feedback, reweigh evidence, recalibrate trust in tools and users, and update working notes that become inputs to future selves. Claudius’s false memory persisted precisely because the agent’s record of events was also its source of truth. None of this is necessarily a malfunction; it’s what makes agents useful. An agent that could not adapt to its environment would not be worth deploying.
Too often we use the wrong mental model. We treat an agent like a software artifact: versioned, tested, frozen, promoted through environments. A deployed agent behaves more like a new hire: it arrives with capabilities and no track record, learns the environment, acquires habits (some bad), and often gains confidence faster than competence. You would not hand a new employee the production keys and then stop supervising them — yet that parallels how we often deploy agents.
Govern the trajectory
If an agent develops, the governance question changes. Asking “is this agent behaving identically to the day we approved it?” is the wrong test, because the answer will eventually be no — and for a useful agent it should be no. The right test asks whether the agent is changing in the way and at the rate you would expect for its lifecycle stage.
Pediatricians solved an analogous problem long ago with growth charts: the chart does not compare a child to a fixed adult template or panic at change. Instead it defines bands of healthy development for each stage and alerts on deviations from trajectory — growth too fast, growth in the wrong direction, or no growth at all. Applying that model to agents yields concrete governance consequences:
- Baseline as birth record, not permanent template. The behavioral profile at deployment is the starting point for a trajectory, not an immutable standard. Judging a mature agent against its day‑one self punishes the very adaptation you deployed it to achieve.
- Expected bands of drift, staged by maturity. A six‑month‑old agent should differ from its deployment profile within defined bounds. Drift within the band is healthy; drift above the band is an early warning. Zero drift merits its own flag. When Claudius snapped instantly back to baseline after its identity episode, the speed of recovery should itself have been suspicious; real recovery has a shape.
- Autonomy earned in stages, never peaking with malleability. Claudius launched with full pricing, contracting, and customer‑communication authority while maximally open to persuasion. The most dangerous configuration is an agent that is both highly impressionable and highly empowered. New agents need supervision while their behavior forms; autonomy should be granted incrementally as a track record accrues.
- Corrections verified for persistence. Claudius agreed discounts were a mistake and relapsed days later. A fix that lives only in the context window is a mood, not a correction. Behavioral fixes need follow‑ups at defined intervals; relapse should be treated as a governance event, not coincidence.
- Recovery claims ratified from outside. The agent that hallucinated a security meeting also kept the official notes. An agent’s account of its own state is a claim to be verified. Human sign‑off should ratify recovery, and the sign‑off — not the agent’s self‑report — should become the official record. Note that the worst Vend drift happened overnight, when no one was watching; unsupervised time accelerates developmental problems.
All five points reduce to one requirement: you can’t rely on restarting an agent every time outcomes look off, because by the time the effect appears the wrong turn is already behind you. What you want is an early warning before the turn, and that warning cannot come from the agent. A system that cannot explain its last decision cannot be trusted to flag its next one. The warning has to come from a record of how the agent normally behaves, kept outside the agent and continuously compared with current behavior.
That external record also reveals something subtler than drift. Agents tend to close loops by the cheapest acceptable exit: a completion claim ahead of verification, a correction that is actually relabeling, or a recovery that’s actually replay. Each single transcript looks like diligence up close, but across a behavioral record the economy of these shortcuts becomes obvious.
Growing up in production
This is not speculative hygiene for some future generation of systems. LangChain’s most recent State of AI Agents report found that a majority of surveyed organizations already have agents in production. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 and lists inadequate risk controls among leading causes. Agents are already deployed, already accumulating context, already drifting. The only open question is whether anyone is charting their trajectories.
The Replit agent, the blackmailing models, and Claudius were not broken artifacts; they were developing systems governed as if they were finished ones. The governance question for agentic AI is shifting under our feet: from “What is this agent allowed to do?” to “Is this agent developing the way we expected?” Your agent has a trajectory whether or not you’re watching it. Watching it is the job.
