Caroline Wang and colleagues at the University of Texas at Austin and Google analyzed how large language models (LLMs) and humans differ in decision-making by studying their play in the classic rock–paper–scissors game. The study aimed to reveal whether LLM behavior reflects the same strategic patterns as human players and whether those behaviors can be represented by interpretable code.
Method: games, data, and AlphaEvolve
The authors ran experiments against 15 preprogrammed bots of varying complexity. Individual LLMs — Gemini 2.5 Pro, Gemini 2.5 Flash, GPT‑5.1, and GPT‑OSS 120B — each played 20 games consisting of 300 sequential rounds. The team also used previously collected records of games between humans and the same bots.
For every player they logged the round-by-round choices (rock, paper, scissors) and the outcomes (win, loss, tie). They then applied AlphaEvolve, an agentic method that iteratively optimizes Python code through an evolutionary process to predict the next move.
AlphaEvolve started from a simple template program provided by the authors. Across an undisclosed number of evolutionary steps, Gemini 2.5 Flash proposed modifications to improve a predictive function that balanced simplicity (measured by Halstead effort) against evaluation likelihood (how well the program predicted a player’s choices).
For each player the researchers selected the simplest program that achieved near-maximum predictive accuracy (within a small margin of the best). Each chosen program produced the highest evaluation likelihood for the player it was evolved to predict — in other words, it represented that player’s behavior better than it did any other player’s.
Results: which models resembled each other?
The team tested the chosen programs on game data that AlphaEvolve had not processed and compared how well each program predicted other players’ moves. They then inspected the programs to infer the strategies each player used.
A key finding was that programs representing Gemini 2.5 Pro, Gemini 2.5 Flash, and GPT‑5.1 predicted each other’s moves nearly equally well when facing bots. For example, when predicting the actions of Gemini 2.5 Pro, the programs for Gemini 2.5 Pro, Gemini 2.5 Flash, and GPT‑5.1 achieved evaluation likelihoods of 0.507, 0.507, and 0.506 respectively. Programs representing human players and GPT‑OSS 120B performed worse at predicting that trio, with evaluation likelihoods of 0.476 and 0.403 respectively, indicating different underlying strategies.
Code inspection suggested that Gemini 2.5 Pro, Gemini 2.5 Flash, and GPT‑5.1 preserved sequential patterns more effectively than humans or GPT‑OSS 120B. The programs modeling those LLMs tracked frequencies of each possible move conditioned on the player’s previous one or two moves — for example, how often a player produced sequences like rock→scissors→rock or rock→scissors→paper over three rounds. By contrast, programs representing humans and GPT‑OSS 120B tracked only the opponent’s most recent move frequency.
A further distinction: programs modeling Gemini 2.5 Pro, Gemini 2.5 Flash, GPT‑5.1, and human players computed a preliminary value for each possible next move based on (i) the possible next move itself, (ii) the bot’s previous move, and (iii) the player’s previous move. The GPT‑OSS 120B program computed values based only on the possible next move.
Why this matters
The study highlights that while LLMs can display humanlike behavior on the surface, their internal strategies may differ and in some cases be more structured, encoding multi-step sequential patterns that typical human players do not. Synthesizing interpretable code from LLM behavior with methods like AlphaEvolve provides a practical window into LLM decision logic, which is otherwise often opaque.
The authors’ observations also suggest that LLMs do not merely mimic human behavior present in their training data; they can organize task-solving into explicit, systematic rules. That points to a form of learning that can differ qualitatively from human heuristics.
