OpenAI’s largest GPT-5.6 variant, Sol, scored 7.8% on the ARC-AGI-3 benchmark. At first glance this number looks strikingly low—human performance on the same test reliably exceeds 90%. At the same time, it represents a substantial jump compared with prior models: GPT-5.5, released three months earlier, scored just 0.43%, and the earlier Opus 4.8 achieved 1.5%.
What ARC-AGI-3 measures and why it matters
ARC-AGI-3 is designed to look deceptively simple to human observers while testing what François Chollet describes as "fluid, not crystallized" intelligence—problem solving in novel situations rather than applying learned routines. Tasks are intentionally out-of-distribution: success depends on on-the-fly reasoning about rules and dynamics the system has not seen during training.
What Sol’s result reveals
The OpenAI team emphasized not only the score or efficiency (the full evaluation at maximum reasoning effort cost nearly $20,000), but also qualitative aspects of Sol’s problem-solving. Key findings:
- Sol is the first verified frontier model to solve an ARC-AGI-3 game; its advantage comes from correctly orienting itself in novel environments rather than from purely better execution.
- The model’s standout skill is scene comprehension: it frequently discovers the core game mechanics that other models miss.
- When Sol fails, failures typically occur downstream in planning or execution rather than at perception; Sol often perceives the relevant mechanics but cannot reliably compose deeper chains of inference into a working plan.
A concrete example from the report: Sol discovered a complex mechanic on the LP85 level where pieces must be "parked" until needed. The description reads: “The horizontal tracks are independent. Park the lower color-11 tile one step right, then the vertical loop can move the upper tile without disturbing it. ACTION6 48 37”.
Why the result is paradoxical
The situation is paradoxical for three reasons:
- By human standards the 7.8% score is very low: humans can infer rules and reach >90% success.
- By AI benchmarks the 7.8% score is a big improvement: GPT-5.6 is roughly twenty times better than GPT-5.5 (0.43%), marking a notable milestone in the ARC-AGI series, while also scoring above 90% on ARC-AGI-1 and ARC-AGI-2 with low cost per task.
- The benchmark itself is unusually informative: the best models can appear excellent on many tasks yet still perform poorly on a simple-seeming test that probes fluid intelligence. This raises questions about what kinds of intelligence current models truly capture.
Implications and open questions
Sol’s performance suggests that contemporary large language models—even ones that can self-supervise further training and tackle difficult mathematical problems—still struggle with rapid, flexible reasoning in unfamiliar, perceptual problem settings. The report’s diagnosis is that improving scores beyond this point may depend less on raw intelligence and more on scaffolding: enhancing memory, planning, and the ability to chain inferences over longer horizons.
The ARC Prize team is already working on ARC-AGI-4, which prompts further questions: how many more ARC-AGI iterations can be designed that are easy for humans but hard for AI? What metrics should determine whether we have reached AGI (artificial general intelligence)? Until such tests are saturated, claims of AGI remain premature.
The author notes a personal judgment: while not convinced AGI has been achieved, they welcome that GPT-5.6 Sol is beginning to penetrate the type of challenge that ARC-AGI is intended to expose.
