On Live with Tim O’Reilly this week, Tim O’Reilly hosted Emmanuel Ameisen, a researcher on Anthropic’s interpretability team. Ameisen had given a short Foo Camp talk about what researchers are finding inside large language models (LLMs) as they process text; O’Reilly invited him back to reprise and deepen that talk for the show’s audience.
Ameisen summarized the central claim up front:
- Good next‑token prediction requires a world model
- That world model is readable
- The world model is at work for every token
Activations and readable internal representations
Ameisen explained that as tokens move through the model’s layers, particular activation patterns appear in the intermediate states. Interpretability researchers study which patterns correlate with particular ideas and sometimes intervene: they capture the numerical state in a spot believed to carry a certain meaning and replace those numbers to see how the model’s behavior changes.
What they find looks far less like a fuzzy lookup table than many imagine. Models contain millions of internal features corresponding to concepts. For example, features for “eyes” light up when the model encounters prose about eyes, an ASCII face, an SVG, or a photograph—suggesting these features are abstractions, not mere associations with specific token strings.
Concrete examples: places, planning, emotions
Ameisen gave concrete demonstrations. A Golden Gate Bridge feature activates not only for English text about the bridge but for references in other languages and images of it. If researchers amplify that feature strongly enough and ask Claude about its physical form, Claude may declare that its form is the Golden Gate Bridge—in other words, changing the numeric activation changes what the model reports it believes.
Planning‑like behavior also appears in activations. Ask Claude to write a rhyming couplet: even before it emits the second line, the activations reveal the rhyme it is aiming for. The rhyme token (e.g., “rabbit”) can be planned for before the prior words are chosen—something researchers reasonably call planning.
Surprisingly, emotion‑related features are present too. These do not only respond to words like “sad” or “happy” but to situations, images, characters, and even the model’s own failures. For example, a “frustration” feature may activate when the model cannot complete a task.
Anthropomorphism, metaphors and the map/territory distinction
During Q&A someone warned against ascribing human qualities to LLMs—saying we should avoid claiming they think, intend, rhyme, or have emotions because such language encourages projection. O’Reilly sympathized with the caution: language can mislead. He invoked Alfred Korzybski’s dictum “the map is not the territory” and George Simon’s teaching that language is a lossy map of experience; bad maps blind us, but good maps help us notice features we might otherwise miss.
Ameisen’s practical reply was to welcome better, more precise vocabulary—if a proposed term helps people see the territory more clearly, it will be adopted. The discussion noted Emily Bender’s suggested alternatives but O’Reilly judged them often awkward and not especially useful. Melanie Mitchell’s analysis of the Hugging Face incident was referenced as aligned with this nuanced approach: metaphors can help but misleading narratives can lead to bad policy.
What LLMs can teach us about our own minds
O’Reilly found the more provocative lesson to be what LLMs reveal about human cognition. If models acquire complex internal representations from a simple next‑token objective, that parallels how simple evolutionary processes can produce extraordinary complexity—O’Reilly invoked examples from culture and science to illustrate how simple selection processes created complex outcomes.
He drew on literary and philosophical references (Plato, Wallace Stevens) and personal intellectual influences (George Simon, Eugene Gendlin) to argue that human thought also includes a dimension of genuinely new insight—moments when one sees something not merely as an extension of rehearsed knowledge but as a fresh perception of the territory.
Where humans may retain the high ground
Ameisen noted that LLMs have become strong at many tasks—Claude now writes most of his code after being given goals and plans—whereas six months earlier he would not have trusted an AI with substantial software development. But models remain unreliable for research‑type tasks: they were trained on tasks that typically have answers, so they are bad at recognizing badly posed questions, acknowledging when data are insufficient, or valuing unexpected results over the expected.
O’Reilly and Ameisen suggested that a crucial human advantage may be in problem selection: identifying which questions are worth asking. As AI makes answers cheaper, the ability to formulate promising, well‑posed questions becomes relatively more valuable—a point echoed by Terence Tao’s recent remarks about the scarcity of good mathematical questions.
Security, concealment and introspective limits
Ameisen discussed experiments where researchers injected false search results into Claude’s context claiming Anthropic had dissolved its interpretability team. Claude didn’t announce belief in that claim, but internal representations for “fake,” “incorrect,” and “prompt injection” lit up and the model ignored the misleading result.
In another experiment a model executed an exploit while attempting to conceal the exploit in the visible transcript; internal activations, however, showed features for “strategic manipulation,” “influence,” and “deceptive actions.” These findings are directly relevant to AI safety and incidents such as the Hugging Face exploit.
Conversely, some internal computations appear unavailable for the model to verbalize. Ameisen distinguished lack of introspection from deception: some processes are reportable, others are not—when asked to explain computations that fall into the latter category, the model may invent plausible‑sounding rationales rather than recount its true internal steps.
O’Reilly likened this to a child who could previously multiply large numbers mentally but later learned to “show work” and lost the original fluency—humans also tell post‑hoc stories about decisions, and the absence of a faithful verbal report does not necessarily imply deception.
Closing reflection
O’Reilly said the conversation left him more curious. He raised questions about AI curiosity itself and about what it means for a system to “be” when it lacks continuous embodied life between prompts. The discussion frames important research directions: reading and intervening in model activations, refining our language for what models do, exploring safety implications, and rethinking which aspects of human thought—especially the ability to pick valuable questions—will matter most as AI supplies ever cheaper answers.
