OpenAI engineers report that enabling two Responses API settings—retained private reasoning and context compaction—substantially improved GPT‑5.6 Sol’s results on the ARC‑AGI‑3 benchmark. On the public task set, GPT‑5.6 Sol scored 13.3% Relative Human Action Efficiency (RHAE) with the official harness; with retained reasoning and compaction enabled it scored 38.3%. The change also reduced output tokens by roughly sixfold.
What ARC‑AGI‑3 measures
ARC‑AGI‑3 is a benchmark of 2D puzzle games designed to evaluate how well agents learn and reason in unfamiliar environments. Agents must explore games and infer rules and mechanics without explicit instructions. Twenty‑five demo games are available at arcprize.org/tasks. RHAE compares model performance to a human baseline; based on official gameplay logs, OpenAI estimates the average human tester scored about 48%.
Why GPT‑5.6 Sol performed poorly with the official harness
ARC intentionally uses a minimal, tool‑free harness to make model limitations more visible and to standardize comparisons. OpenAI’s investigation identified two harness design choices that hindered GPT‑5.6 Sol:
- After every game action the model’s private reasoning messages were discarded. That meant the model could not retain its internal plans or chain‑of‑thought between turns; it only saw past moves and short notes, not the thoughts that generated them.
- The harness used rolling truncation: when the conversation exceeded a character limit (the ARC implementation used 175,000 characters), oldest messages were dropped. As a result the model lost both private reasoning and older observations/actions as context grew.
Together, these behaviors forced the model to re‑interpret the game from scratch each turn and impeded learning across longer runs.
How OpenAI changed the harness
To better match production usage in ChatGPT and Codex, OpenAI reimplemented the ARC‑AGI‑3 harness using the Responses API and applied two settings:
- Retained reasoning: passing the previous response ID causes the API to keep private reasoning messages in the conversation history, so models can access their prior internal thoughts across tool calls and turns.
- Compaction: instead of rolling truncation, compaction compresses older context to preserve important information rather than simply deleting the oldest messages. OpenAI notes their implementation uses a 175,000 token limit (which is similar to the 175,000 character threshold in practice for action grids).
Effects of retained reasoning and compaction
With retained reasoning, GPT‑5.6 Sol spent less time deliberating before each action because it no longer had to reconstruct the game state from scratch each turn. With compaction, the model preserved what it had learned about each game across longer runs and operated with fewer output tokens. Combined, these two changes enabled faster decisions, more coherent strategies over time, and roughly a threefold increase in RHAE with a sixfold reduction in output tokens on the public ARC‑AGI‑3 set.
Broader takeaway for evals and developers
OpenAI emphasizes that evaluations rarely measure a model in isolation: scores also reflect choices about API settings, harness design, and prompting. They note prior surprises where low public eval scores were traced to generic harnesses that dropped reasoning messages. For API developers seeking best performance, OpenAI recommends using the same retained reasoning and compaction settings used in ChatGPT and Codex. For model comparisons, they recommend evals that adopt these settings since they better match real‑world deployments.
Credits and how to try it
OpenAI thanked ARC for their long‑term work on AGI evaluation and for the analysis that prompted this follow‑up. Anyone who wants to test frontier models against the public tasks can try the games at arcprize.org/tasks.
Key numbers: GPT‑5.6 Sol RHAE on public ARC‑AGI‑3: 13.3% (official harness) vs. 38.3% (retained reasoning + compaction). Estimated human average RHAE: ~48%. Context limit referenced: 175,000 (ARC used characters; OpenAI references 175,000 tokens in their implementation). Output tokens decreased by ~6x.
