The role of puzzles and what they reveal about AI
Puzzles and games have been used as benchmarks for artificial intelligence since the field’s early days. Just as people test their reasoning with crosswords or logic puzzles, researchers have long measured AI progress with game-like challenges. The term “machine learning” was popularized by IBM computer scientist Arthur Samuel in 1959 in connection with an algorithm that learned to play checkers; chess and Go have similarly served as prominent testbeds.
In recent years, models’ puzzle-solving abilities have improved rapidly. For example, a Columbia University team showed in late 2024 that even top models could solve only about 18% of the New York Times Connections puzzles; by early 2025 some models were solving those puzzles nearly perfectly.
Puzzles do more than track capability gains: they expose specific strengths and weaknesses. Current systems still err—small edits to classic puzzles often mislead them, and visual problems are a notable weak spot.
Spatial reasoning
One domain where humans retain a substantial advantage is spatial reasoning. Mental-rotation tasks common in IQ tests ask whether different images show the same object from different angles. Although today’s language models often accept and analyze visual inputs, they generally fail at these problems. Despite discussions about “world models” helping AI understand physical environments, large language models (LLMs) still do not appear to manipulate 3D objects the way spatial experts such as architects or mechanical engineers do.
Memory and adaptability
Frontier LLMs have enormous memories: they were exposed to vast quantities of information during training and can reproduce many facts verbatim. That helps in trivia-style tasks but can be a liability when a puzzle closely resembles material from the training set. Models may overlook crucial differences and produce memorized answers instead. A 2024 study by researchers at Google and the University of Illinois Urbana-Champaign trained and tested models on slight variations of Knights and Knaves puzzles—where some characters always tell the truth and others always lie—and found models often tripped over minor alterations that humans detect. The SimpleBench benchmark shows a similar effect: because its questions resemble more complex problems that models likely saw in training, humans tend to spot the trick while even advanced models fail.
Abstract and visual reasoning
Models stumble not only on 3D visual tasks but also on two-dimensional problems. This limitation strongly influences performance on the ARC-AGI benchmark, which asks solvers to infer abstract, general rules from examples. Research indicates that even when models answer ARC-AGI items correctly, they frequently rely on convoluted, non-generalizable internal rules, whereas humans use simpler visual concepts. Nonetheless, models have made substantial progress on ARC-AGI over the past year, but certain puzzles—like the example discussed in the original piece—still defeat them.
Intuition
Humans have cognitive biases that models do not necessarily share. Psychologists have designed problem sets that invert the SimpleBench phenomenon: in these tasks, people often give knee-jerk responses while models respond deliberatively. Some items exploit intuitive errors in the way we do arithmetic; others are worded so an obvious but incorrect answer suggests itself unless the question is read carefully.
Scaling complexity
In some problems, model success is a matter of scale. An Apple study found LLMs can master simple versions of the Tower of Hanoi (moving disks one at a time without placing a larger disk on a smaller one) and basic river-crossing puzzles, but performance degrades once the number of disks or people reaches six or more. Another investigation by researchers at the University of Washington, Stanford University, and the Allen Institute for AI observed similar limits for logic grid puzzles, where solvers deduce attributes of individuals from a list of clues. The Apple paper attracted wide attention, but commentators debated whether these results indicate a unique reasoning limitation of LLMs or simply reflect that all systems make more mistakes as task complexity increases.
What these gaps tell us
Different puzzle types—mental rotation, memory-and-adaptivity tests, abstract visual problems, intuition traps, and scaling challenges—illuminate distinct aspects of human versus model cognition. Those failure modes are not merely academic: they point to where systems need improvement and inform expectations for deploying models in real-world situations.
Author and credits
The original article was written by Grace Huckins, an AI reporter at MIT Technology Review with a PhD in neuroscience. The tasks and illustrations discussed draw on multiple sources: spatial-reasoning research by Stogiannidis, McDonagh, and Tsaftaris (Mind the Gap, 2025); Knights and Knaves material courtesy of Dan MacKinnon; SimpleBench Team (2024); ARC Prize Foundation; Hagendorff et al. (2023) for the lightning-round work; an adapted river puzzle from Alcuin of York (ca. 800 CE); and ZebraLogic (Lin et al., 2025). Some images and materials are licensed CC BY 4.0 or under other open licenses as noted by those sources.
