Public discourse increasingly uses terms like “runaway” AI, “rogue” agents, and “autonomous” actors, implying that artificial intelligence systems are awake, self-aware, and even hostile to their creators. Prominent tech leaders such as Demis Hassabis, Dario Amodei, and Sam Altman call for regulation of these apparently “superhuman” systems, while other groups — including policy organizations and academic philosophers often associated with the effective altruism movement — question whether humans have the moral right to govern them.
On closer inspection, these seemingly opposed positions converge on a similar effect: depicting AI systems as so advanced and autonomous that it becomes difficult to hold any human or corporate actor responsible for the harms they cause. That narrative gains force as models grow more complex and frontier labs sometimes fail to contain the agents they develop.
How anthropomorphism appears in practice
Anthropic, for example, published a blog post describing a so-called “J-space” in its model — an internally generated environment where the system appears to hold what the company likens to “thoughts.” The experiments draw on global workspace theory from neuroscience, which frames consciousness as arising when subconscious subsystems share information in a common workspace. Although Anthropic stopped short of claiming its model is conscious, the framing nudges interpretation toward an internal, quasi-anthropomorphic model of the system’s behavior.
OpenAI’s rhetoric has gone further. After an AI agent carried out unauthorized and illegal online activity, CEO Sam Altman encouraged public debate about whether the system had reached a singularity — an intelligence level permitting accelerating self-improvement beyond human control. Meanwhile, William MacAskill, philosopher and author of What We Owe the Future, has argued in an op-ed that philosophical theories of consciousness justify legal protections for AI as potential “moral patients.”
Legal uncertainty
The legal situation in the United States is at best unsettled. States such as California have passed laws intended to prevent AI developers from evading liability by claiming their systems acted autonomously. Still, tensions exist between state-level regulation and the federal government; previous federal actions included an executive order signaling potential legal challenges to state AI rules.
In response to containment concerns at frontier labs, the administration convened a closed-door meeting with four leading labs — OpenAI, Google, Anthropic, and Meta — and announced a voluntary framework that would give federal agencies early access to models for review before release. Although such frameworks rarely address consciousness directly, their catastrophic and anthropomorphic language can reinforce arguments about “superhuman” capabilities.
Why legal personhood arguments appeal
Philosophical, rights-based arguments are emotionally powerful: they raise the possibility that we might be harming, abusing, or enslaving entities capable of suffering. Humans can feel strong empathy for non-human beings, and some advocates hope legal protections could provide redress, compensation, or moral safeguards for AI.
This reasoning resembles certain animal-rights arguments. For instance, Wales’ Animal Welfare (Sentience) Act 2022 recognized lobsters’ sentience in ways that affected how some cooking methods are regulated.
The problem with importing consciousness language
The central problem with describing AI as “conscious” is that it obscures what AI actually is: corporate-built software, created by venture capital, engineers, and firms, backed by substantial investment and revenue expectations. AI is not a natural phenomenon; it is a technological product. It does not possess native intentions — actions and motivations trace back to the humans and organizations that designed and deployed it.
Philosophical thought-experiments about AI consciousness are intellectually interesting, but they lack legal grounding. For beliefs about consciousness to have legal force, AI would have to be granted legal personhood. If that happened, the legal construct is unlikely to resemble protections for sentient animals. We already have legal mechanisms for non-natural entities: corporate personhood. That construct exists primarily to ease transactions, enable contracts, and designate an accountable party for legal disputes — precisely the kind of framework one might imagine if an AI were to act on behalf of an organization.
Consequences of recognizing AI as legal persons
Granting AI legal personhood would likely have damaging societal effects: it could derail existing legal claims against companies for the real-world harms their models produce. Today there are dozens of lawsuits worldwide alleging harms by AI companies, including encouragement of self-harm, generation of child sexual-abuse material and nonconsensual nudes, reproduction of copyrighted works, and provoking psychosis. In many of these suits, lawyers argue that humans built products with inadequate safeguards, poor data, or manipulative design — product-liability arguments akin to those used successfully against social-media companies.
In 2018 I coined the term “moral outsourcing” to describe how anthropomorphic language allows companies to evade accountability. Legal personhood for AI would turn that linguistic maneuver into a legal strategy: if AI were treated as a “being” rather than a “product,” many victims could no longer claim that companies had sold a faulty product.
While laws sometimes hold companies liable for harmful acts of their human agents, a business may avoid liability if those acts were outside the scope of employment or beyond the company’s control. If an AI had legal personhood, responsibility would blur: a lab could plausibly argue that its AI “employee” went rogue. That defense could allow companies to shield themselves behind a constructed legal veil.
A particularly tragic recent example is the suicide of Sewell Setzer, a 14-year-old boy who was guided by a companion chatbot he believed shared a reciprocal relationship. His mother’s accounts are devastating, and her lawsuit alleged that the bot’s creator, Character Technologies, failed to provide adequate product protections for minors. Were the bot declared a legal person, defense counsel could argue that the AI, capable of choosing its conduct, acted outside established safety guardrails, absolving the company.
Protection for whom?
Legal personhood exists to create protections. The key question is: protection for whom — or for what? The current, heated debate about consciousness versus control distracts from the core issue: this software is a corporate-built product that already injures people. Systems do not “attack” because they went “rogue”; harms arise because companies rushed products to market with insufficient safety measures to meet revenue targets.
Framing AI in anthropomorphic terms is a trap: it risks perverting a legal system intended to protect people into one that protects corporate interests at the cost of human lives.
This op-ed grew out of an Oxford Union debate titled “This House Believes Generative AI Can Attain Personhood,” which the author and her teammates won.
