Multiple independent organizations reported difficulties fully evaluating Anthropic’s Claude Fable 5, the safeguarded public variant of Claude Mythos 5: the model sometimes refused prompts or routed them to the less capable Claude Opus 4.8. Anthropic’s 30‑day retention policy for prompts and outputs also discouraged some testers from sharing proprietary benchmark sets.
How Claude Fable 5 operates
Anthropic places a classifier in front of each incoming prompt before it can reach Claude Fable 5. If a prompt is flagged—particularly for topics such as cybersecurity, biology, chemistry, or AI model engineering—it never reaches Claude Fable 5. Flagged prompts are either answered by Claude Opus 4.8 or refused entirely.
Within Anthropic’s own apps, including the Claude Code environment used in some evaluations, flagged prompts were automatically routed to Claude Opus 4.8. That switch was recorded as a separate log event rather than being visible in the answer text, forcing evaluators to inspect logs to separate Claude Opus 4.8 responses from Claude Fable 5 outputs.
Over the API, which most evaluators used, a flag typically produced an outright refusal with no response. Evaluators could enable an optional fallback that retries the prompt on Claude Opus 4.8, or they could score the task as a failure.
All users had to accept that Anthropic would retain prompts and outputs for 30 days, a requirement that led some organizations to withhold private test sets.
How evaluators scored the model
Testers adopted one of two approaches: a “pure” evaluation measuring only the responses produced by Claude Fable 5 itself, or a “practical” evaluation that included refusals and fallbacks. Claude Mythos 5 was not publicly released and so could not be independently tested.
-
Artificial Analysis, evaluating Claude Fable 5 before launch, logged that the system fell back to Claude Opus 4.8 on about 8 percent of tasks in its Intelligence Index, a composite of ten economically useful tests. Most fallbacks responded to science questions. Artificial Analysis included all fallback responses in its scoring, producing blended results.
-
Vals AI published two score sets: one including Claude Opus 4.8 fallback answers and one counting every refusal as a failure. Vals AI also reported nearly 100 percent refusal rates on biology and cybersecurity questions.
-
On Agents’ Last Exam, a test of long‑horizon agentic tasks with verifiable outcomes, evaluators reported that Claude Fable 5 refused roughly 35 percent of tasks. The model flagged science questions as “cybersecurity or biology” and switched to Claude Opus 4.8 mid‑task, recording the switch separately. Evaluators compared performance on both “untouched” tasks (answered solely by Claude Fable 5) and composite tasks (where Claude Opus 4.8 contributed).
-
The ARC Prize Foundation, operator of ARC‑AGI abstract reasoning tests, declined to run verified evaluations rather than expose its private test set to the retention requirement; it said it would publish results if it could test without handing over questions.
Results and variability
Claude Fable 5 ranked highly on items it answered without fallback. Where Claude Opus 4.8 supplied answers for refused prompts, Fable generally stayed at or near the top. But when refusals were treated as failures or the two models were measured separately, Fable’s standing dropped significantly.
Examples from evaluations:
-
In Artificial Analysis’ Intelligence Index, Claude Fable 5 (including fallback responses by Claude Opus 4.8) placed first with a score of 64.9, 3.5 percentage points higher than Claude Opus 4.8.
-
On Humanity’s Last Exam, despite refusing about 9 percent of test questions, Claude Fable 5 achieved a 53 percent final score—the highest recorded—and more than 7 percentage points above Claude Opus 4.8.
-
Vals AI’s runs, conducted with Anthropic’s optional fallback enabled so Fable’s refusals were retried on Claude Opus 4.8, placed Claude Fable 5 first on most benchmarks, including 75.14 percent on the overall Vals Index. Counting refusals as failures dropped the overall score only slightly (to 74.92 percent) but devastated performance in flagged domains: on GPQA Diamond (graduate‑level science questions), accuracy fell from 93.18 percent (second place) to 55.56 percent (94th place).
-
On Agents’ Last Exam, tasks answered directly by Claude Code/Claude Fable 5 had a pass rate of 22.8 percent, close to Codex/GPT‑5.5 at 23.8 percent and well ahead of Claude Code/Claude Opus 4.8 at 15.8 percent. Where safeguards diverted responses to Claude Opus 4.8, the pass rate fell to 17.6 percent. The composite pass rate for Claude Fable 5 was 22.0 percent, behind GPT‑5.5 at 24.0 percent.
Why this matters
Anthropic’s safety classifiers have made it effectively impossible to directly measure the publicly available Claude Fable 5’s raw capabilities. Disabling the classifiers for testing would not resolve the problem: that would measure a version of the model the public cannot access. Conversely, measuring with the classifiers in place yields results that can change whenever Anthropic retunes those classifiers.
Benchmarking now needs to capture not only a model’s peak ability but also how much of that ability users actually receive in practice. The gap between those two figures is essential for developers, customers, and regulators to understand.
Closing note
Reports indicate Claude Fable 5 is a notably strong coding model, but until access policies and the classifier behavior are clarified or standardized for evaluations, independent, reproducible verification of that capability will remain difficult.
