On October 15, 2026, Tim O’Reilly hosted Andrew Odewahn, CTO of O’Reilly Media, for an hour-long Live with Tim O’Reilly episode. Over the past year Andrew’s engineering team built Expert Intelligence: a suite that links an AI research agent to an organization’s internal systems, gives it a skill describing how to reason about those systems, and grounds its outputs in practitioner frameworks from the O’Reilly library.
Building the research agent
Andrew said the research agent grew from a personal need: questions like how long a migration from RabbitMQ to Pub/Sub would take. He built the agent in four steps: map the organization’s information systems as a hierarchy, connect them via MCP, write a skill describing how the agent should use them, and add the Expert Intelligence layer.
The map of sources of truth made the rest possible. Strategic goals and feature priorities live in Productboard; those become epics, sprints, and stories in Jira; stories become code in GitHub. O’Reilly’s platform is described by a service catalog of roughly 300 services and owners, and observability tools report service performance against SLOs. The skill also encoded local conventions (for example, some teams mark an epic finished by end date, others when every ticket is closed). The agent was explicitly instructed to be a research agent, not a coding agent, and to produce reports.
Connecting systems via MCP, Andrew said, was “the most completely tedious step.” Productboard had no MCP server at the time, so he wrote a wrapper. He recommended other organizations start from their vendor list: export procurement data, ask the model which vendor products are likely to hold relevant information, and order them. O’Reilly product managers plan to turn this into a skill for an Expert Intelligence Skills Hub.
From a hiring case to a toil audit
The central example was a before-and-after analysis Andrew had previously written up. An engineering manager asked for a hire; the initial agent report described a five-person content-and-metadata team, the ten tier-A services it owns (plus B and C services), the Pub/Sub migration at 64% complete, and the full backlog. That report concluded the team had a capacity problem and probably needed to hire — useful but perhaps overly detailed.
When Andrew connected the agent to Expert Intelligence and asked it to act like a consultant, the second report reached a different conclusion: the team’s operational overhead was structural, not a staffing deficiency. The agent imported the concept of toil from Google’s Site Reliability Workbook, using it to separate backlog items into toil (manual, repetitive, automatable work) and the work engineers wanted to do. It recommended a structured toil audit before the next planning cycle, an explicit toil-reduction target, and treating runbook ownership as a first-class engineering artifact. The team followed the recommendations and closed a substantial portion of the backlog that had been a cognitive burden.
The report also cited Jim Highsmith (signer of the Agile Manifesto, author of Wild West to Agile) to argue that technical debt was compounding. Andrew emphasized that the system did not only offer abstract frameworks but placed them on a specific backlog item — for example, the Django upgrade — producing actionable guidance.
A second analysis examined services crossing team boundaries. Drawing on microservices experts and Kent Beck’s coupling work, the agent diagnosed the video metadata service as a complicated-subsystem problem rather than a boundary problem. In that run the agent pulled quotations, attributed them, and linked to source text; the analysis was effective and led to retiring a couple of systems. Andrew said: he doesn’t want LLMs adding code — he wants them reducing it.
Andrew was surprised by colleagues’ reactions. Early reports were data dumps without clear provenance, making them less actionable. When the analysis cited known experts, documents became “much more approachable, much more human,” and people engaged more deeply.
Tim O’Reilly: invoking experts as a focusing mechanism
Tim put the demonstration in context by arguing for the return of expertise rather than the idea that models can be left to ‘‘rip’’ on their own. He cited Felix Rieseberg of Anthropic’s post “How to Be More Ambitious: A Tutorial for Humans with Models,” which describes four ambitions when working with models: speed, altitude, depth, and range. Most economic arguments about AI focus on speed; in Tim’s reading Andrew’s work added altitude (thinking about outcomes) and depth (approaching problems like a field-leading expert) by invoking experts.
Tim offered a metaphor he has used before: AI programming as spellcasting. Andrew’s approach is like saying, “By the power of Kent Beck, think about this problem.” Even when Andrew didn’t name a person (for example, the SRE workbook), pointing the model at the O’Reilly publishing canon raised the bar for outputs.
Andrew said his first prompt asked the model to “be snobby” and look for canonical works (for example, Designing Data-Intensive Applications) and adopt those ways of thinking rather than accept the model’s first impulses.
What models do without direction
Tim argued models don’t necessarily do their best work unprompted: they can latch on to a random file instead of canonical sources. Andrew told a recent story: he had the agent review incentive plans and it returned a framework from a company called Robusto — because he hadn’t connected to the VPN. Once connected, the agent produced results citing Harvard Business Review. The model had keyed on the first thing it found.
Tim raised an economic point: some systems (he cited Anthropic’s Claude) limit how much can be quoted from sources, which he reads as a way companies avoid paying for content. The result is that models can’t always draw on the best material in specific ways and fall back to generalities. O’Reilly takes a different approach: it applies a royalty model (as with O’Reilly Answers). Material returned through the Expert MCP server can be cited and authors are paid royalties, so expert content has a market return.
Expert Intelligence is in prerelease with founding partners while O’Reilly refines its economics; interested organizations can request access via the product page.
Putting discussion where the AI can see it: SuperAnswers
The presentation’s second part addressed what happens after an agent writes a report: decisions require consensus and humans must verify that the model didn’t anchor on irrelevant or outdated sources. Initially the team circulated Google Docs, but two problems emerged: the most valuable part of a document under review is the comment thread, and O’Reilly policy did not allow MCP connections to Google Docs, so the agent couldn’t see the comments; and folding the discussion back into the document wiped out the comments.
Doug Hogan, an O’Reilly engineering director, built an alternative on top of Git and GitHub that Andrew called SuperAnswers. Documents live in a Git repository, discussion lives in GitHub Discussions, and every change is versioned. Andrew can ask the agent for the consensus on a project, have it incorporate comments into a new draft without losing history, and later reconstruct when a decision was made and what people knew at the time. The same record generates derivative documents (scope statements, meeting agendas) and a graph view clusters topics and flags open decisions.
Andrew said he would like to open-source SuperAnswers; he described it as a thin layer on top of Git. He also noted that fragmentation is not limited to engineering: sales teams have data spread across Salesforce, 6sense, and Gainsight, so every function needs to pull those threads together.
Audience questions and closing lessons
Audience members raised concerns that workers might talk to AI instead of each other and that work could disappear into an “AI hole.” Andrew argued that keeping material in Git provides a central place for humans and AI to find and reconstruct past work. He also emphasized AI’s role in enabling new human questions previously too costly to research.
Andrew admitted he hadn’t thought early enough about how people would review and respond to AI-generated documents. People often dismiss an entire AI report if it contains a single factual error. He stressed that AI tools should marshal and present evidence to support human analytical capacity, not replace it.
Tim connected this to deskilling risks. He described a fine-tuning attempt with Fable that failed because the model didn’t guide him through what made a good training dataset. By contrast, O’Reilly Tutor (in development) directed him to relevant O’Reilly material and helped him see his mistakes. Tim argued people have an obligation to keep learning alongside tools.
An audience member seeking to reskill asked how to build depth; Andrew suggested choosing a project and working through it end to end, using AI as a tutor while keeping a canonical book open or attending instructor-led sessions.
The discussion closed on a distinction between AI that makes organizations faster and AI that makes them better at deciding what to do. Andrew summarized that becoming smarter at decision-making requires three things models don’t provide alone: a map of where organizational knowledge lives, a body of trusted expertise, and a place where humans can argue with and respond to AI outputs.
Where to watch
Full episodes of Live with Tim O’Reilly, including this one, are available on the O’Reilly learning platform, YouTube, and wherever you get podcasts.
