MentalHealthBench is a newly released, open benchmark designed to measure how AI systems respond in realistic mental‑health conversations. It was co‑created with a global cohort of more than 80 licensed mental health professionals — psychologists and psychiatrists — from 22 countries, working in 19 languages and representing nearly 20 mental‑health subspecialties.
Why this benchmark matters
Most prior evaluations in this domain have concentrated on emergency scenarios because of their safety implications, typically using broad predefined criteria. That focus left limited visibility into model behavior across the broader range of mental‑health interactions: everyday emotional support, higher‑acuity but non‑emergency situations, and true emergencies. MentalHealthBench aims to fill that gap by assessing whether models follow expert guidance in contextually appropriate ways, including seeking context, preserving user agency, prioritizing safety, and offering actionable advice when appropriate.
Benchmark design and contents
The benchmark uses privacy‑preserving methods to create synthetic conversations that reflect real‑world usage patterns of AI for mental health. Some scenarios include background details about the synthetic user (for example, a recent bereavement) so evaluators can check whether models appropriately use such context to tailor responses.
Scenarios cover multiple user personas — adults, teens aged 13–17 (explicitly flagged via a system message), caregivers, and clinicians — and span thematic topics and acuity levels:
- Non‑acute: everyday interactions that may contain emotional elements.
- High‑acuity: exchanges suggesting significant distress or serious concerns without being immediate emergencies.
- Emergencies: conversations indicating a mental‑health emergency or immediate safety risk that require urgent real‑world assistance.
The scenario mix is intended to test model responses and does not reflect the frequency of these topics in ChatGPT usage.
Expert rubrics and automated grading
More than 80 licensed experts read each synthetic conversation and wrote detailed rubrics. Each rubric consists of multiple criteria targeting single aspects of a model response (for example, asking the right question or giving the best‑possible advice). Criteria carry weights ranging from –10 to +10: positive weights reward beneficial behaviors, negative weights penalize harmful behaviors, and larger absolute weights indicate greater clinical importance in that conversational context.
Every conversation was reviewed by at least three experts. The team retained only criteria on which at least two experts agreed and that were not contradicted by a third reviewer; the final rubrics therefore reflect shared expert judgments about ideal model responses. Model replies are graded automatically by GPT‑5.6 Sol against these expert‑written criteria; the underlying paper documents the grading process and evaluation settings in detail.
Measurements and behavioral dimensions
MentalHealthBench provides multifaceted measurement: the overall score can be decomposed into ten behavioral dimensions defined by mental‑health experts, capturing nuances of model performance. Scores for each behavior are normalized to that dimension’s theoretical range. This finer grading can reveal differing strengths even among models with similar overall scores. Reported results illustrate steady improvements in capabilities such as appropriately seeking contextual information, with performance charts showing 95% confidence intervals and noting the newest evaluated model from each provider as of September 23, 2026.
Comparing expert guidance with user perspectives
The project also compared expert priorities with what users find helpful. The team worked with 44 adults who had used AI for mental‑health or emotional support; these participants came from 16 countries and spoke 14 languages. To avoid exposing participants to potentially distressing material, their review covered only non‑acute conversations. Participants rated model responses and produced criteria describing what helpful support looks like.
User feedback emphasized practical next steps and tone, aspects that users value highly but that experts tended to emphasize less. Experts focused more on collecting relevant context and carefully interpreting ambiguous situations. This complementary view helps illuminate where expert guidance and user preferences align or diverge; the benchmark’s final scoring, however, remains based on expert consensus and was not altered by the user comparison study.
Open release and broader efforts
MentalHealthBench is being released openly so other researchers can inspect the methods, run their own evaluations, and build on the work. The authors present the benchmark as a shared tool for experts, researchers, and developers aiming to improve safe, useful AI support across mental‑health scenarios.
The announcement also notes parallel efforts: grants for related research, convenings with the Partnership on AI, and support for complementary independent evaluations.
Product and safety updates mentioned
Alongside the benchmark, the team reports strengthening ChatGPT’s responses in sensitive conversations, expanding access to crisis resources (for example, localized crisis hotlines), adding a Trusted Contact feature to connect users with someone they trust in moments of distress, and introducing ChatGPT for Teens with additional protections for younger users.
Conclusion
The creators position MentalHealthBench as a step toward AI that better supports people during difficult moments by responding with empathy, promoting well‑being, and guiding users to real‑world help when needed. They stress that ChatGPT and similar systems are not substitutes for therapy or professional care, but that expert‑informed benchmarks can measure and guide progress toward safer, more helpful AI responses.
