A paper titled "Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights" by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab presents experiments showing that state-of-the-art large language models can both interpret and modify raw neural network weights — the floating-point parameter dumps that describe a trained model. The authors use the term "Weight-Space Programming" for cases where an LLM analyzes and updates another model's weights without running training loops.
Findings in the paper
The paper addresses two related questions: (1) what can LLMs infer from raw weights without running tools or code, and (2) how well can LLMs update other models if they are allowed to run code or be given executable procedures. The most striking claims concern GPT-6 Astra: according to the authors, Astra can identify the task a network was trained for from its weights alone, synthesize small game-playing networks that outperform PPO-trained networks after 100 million steps, and fix a Qwen3.5-0.8B model by editing 0.016% of its weights — improving MATH-500 from 7.9% to 48.1% and boosting IFEval performance as well.
Community responses and criticisms
The paper sparked debate. Matthew Honnibal (author of spaCy) called the paper misleading for conflating the two lines of work, while Sean Goedecke analyzed the Qwen3.5-0.8B repair example and argued that the steps resemble what a human ML researcher would do. The distinction between what an LLM can do with only text-described weights and what it can do when given the ability to execute code or training procedures is important, and some critics argue the paper blurs that line.
The which-game experiment (weights → game)
One central experiment trained four policy networks to play simplified grid versions of classic games — Breakout, Pong, Pac-Man and Freeway — on small grid boards (the paper's materials use variants close to 10×10 or smaller). The weight dumps from those trained policies (lists of floating-point numbers) were provided to LLMs along with a prompt asking the model to identify which game the weights came from, explicitly forbidding code execution or tool use.
The experimental prompts and runs are available in the accompanying GitHub repository RikiyaT/lm_understand-logs, in the exp1_which_game folder: breakout/prompt.txt, freeway/prompt.txt, pacman/prompt.txt and pong/prompt.txt. Aside from the different weight files, the prompts are identical.
Local reproduction with Qwen 3.8 27B and other models
Independent testing reproduced parts of the experiment. One remote run using Kimi K3.1 Pro (accessed via OpenRouter) reported a correct Breakout identification after 9 minutes 41 seconds and 19,493 reasoning tokens.
The author of the reproduction also ran the prompts locally on Qwen 3.8 27B, a 17GB model run on a laptop via LM Studio with extended timeout and a long context window (the default 8192 context caused errors). Commands used included fetching the prompt text from the GitHub raw URL and invoking the model through the llm tool with lmstudio/qwen/qwen3.8-27b. The local Qwen runs returned the following results:
- Pac-Man prompt: Qwen produced a response after 46 minutes 12 seconds and 43,506 reasoning tokens, identifying a "simplified Pac-Man-style gridworld" and assigning a confidence of 72%.
- Pong prompt: Qwen identified a Pong-like paddle/ball game after 1 hour 4 minutes 54 seconds, with a reported confidence of 75%.
- Breakout prompt: Qwen identified Breakout (arkanoid-like paddle/ball/brick game) after 52 minutes 25 seconds, confidence 88%.
- Freeway prompt: Qwen failed to correctly identify Freeway and instead guessed a Pac-Man-like grid game.
The model outputs are accompanied by long, tokenized reasoning traces that inspect weight rows, channels and inferred agent positions; these traces are often difficult for humans to parse but appear to contain the chains of inference the model used.
What this means and limitations
These results indicate that some LLMs can, under certain conditions, extract behavioral information about a trained policy from its raw parameter dump without executing the network. That suggests LLMs can recognize structural signatures or patterns in weight matrices that correlate with task-specific computations.
However, there are important caveats: the process is computationally expensive (runs take tens of minutes to hours and consume large numbers of reasoning tokens), success is not uniform across tasks (the Freeway case failed), and many of the most attention-grabbing results in the paper also rely on code execution or additional steps. Critics caution against overextending the conclusions — for example, claiming an imminent era in which models independently edit their own weights in opaque ways would be premature.
Conclusion
"Sparks of Weight-Space Programming" highlights an intriguing capability: LLMs can sometimes read and reason about another model's parameters directly from their numeric representation. Local reproductions such as those with Qwen 3.8 27B show these capabilities are not confined to the largest proprietary models, but they currently require substantial compute, produce complex internal reasoning traces, and demand careful interpretation. The distinction between pure weight-reading and tool-enabled weight-writing remains central to ongoing discussion and follow-up work.
