Beth Israel Deaconess Medical Center in Boston ran an eight‑month clinical trial of Google’s AMIE (Articulate Medical Intelligence Explorer), a chatbot designed for clinical reasoning and diagnostic support. Published in The Lancet, the study evaluated whether the system could safely and usefully conduct pre‑visit conversations with patients about their symptoms and produce summaries for physicians.
Method and participants
Nearly 100 patients participated in the trial. Before their clinical visit, patients consulted with AMIE, which operates on an enhanced version of Google Gemini. The chatbot collected information on symptoms, possible causes and treatment options, and then generated a summary report for the treating clinicians to review prior to the encounter. Trained physicians monitored all conversations in real time and could intervene if needed.
Key findings
- Primary care clinicians judged the chatbot useful in about 75% of cases.
- In over half of encounters, clinicians felt the technology could have meaningfully affected the course of care.
- Across the several hundred conversations, the system produced a single ‘‘hallucination’’: it misdated a past event as occurring in the future.
- In five cases physicians appended additional explanations to the chatbot’s responses, for example when a potential cancer diagnosis was listed among differential considerations.
- The research team never had to halt the system during the trial.
Peter Brodeur, a resident at the hospital, noted that colleagues compared the bot’s performance to that of a well‑trained medical student or resident. Patients generally trusted the system, largely because they knew a live physician would take over care after the AI’s assessment. Adam Rodman, the internist leading the trial, emphasized that no patient would have relied on the chatbot alone without an overseeing specialist.
Why this matters
The Boston trial focused primarily on the safety of a clinician‑oriented chatbot in a real clinical pathway. Its results suggest that AI‑assisted pre‑screening can be a useful adjunct to clinical workflows when used under physician supervision. Following the study, Google has launched a nationwide clinical trial in the United States to compare the effectiveness of AI‑augmented visits with traditional medical appointments.
Safety considerations
Because clinicians monitored conversations in real time, they were able to step in if the interaction became unsafe or emotionally harmful. That supervision likely contributed to the fact that the trial required no shutdowns and revealed only one factual error across hundreds of exchanges.
The study’s detailed findings are available in The Lancet; the Boston experiment represents an early, controlled assessment of integrating AI into pre‑visit patient screening.
Lead image: illustrative.
