On August 23, Surya Narreddi published a short video showing watercolor-style images painted by a language model. The model emits JavaScript that uses the p5.brush library — a toolkit that “adds natural drawing tools to p5.js”. The video went viral quickly and had over 1.5 million views at the time of writing.
Narreddi published a blog post describing an earlier, narrower training stage (close-up flowers rather than full compositions), but not all artifacts were open at that time. He has promised a full technical report. The reproduction described here aims to implement the recipe openly, publishing every piece: the reference pool dataset, the RL environment, training scripts and trained models.
The whole pipeline runs on Hugging Face: training in HF Jobs, the RL environment and the HPSv3 scorer as Spaces, the pairwise judge via Inference Providers, and every artifact on the Hub collected in a single repository. With the two Spaces duplicated, the author provides a single hf jobs command line to launch a GRPO training run on Qwen/Qwen3.5-35B-A3B with LoRA, bf16, gradient checkpointing and a 110-step schedule used for the longer experiments.
Why people liked it
The paintings look loose, imperfect and handmade in contrast to the statistically average, “perfect” images many image models produce. Rather than producing pixels directly, the model writes ~150 lines of JavaScript that use p5.brush to paint. That output is readable, editable and re-runnable, so the decision behind each brushstroke is visible. The model is restricted to ten allowed p5.brush methods, which helps keep a watercolor look.
Rewarding taste with RL
Most RL work for language models uses verifiable rewards (correct answers, passing tests). This project is closer to RLHF: the reward is aesthetic preference, not a single correct answer. The reproduced reward function had four terms with weights the author reports Narreddi converged on:
- gate: 0.05 — sketch compiles, paints something, and does not cheat
- length: 0.05 — a soft push toward longer code
- pairwise judge (Qwen3-VL-30B-A3B-Instruct): 0.60 — style judged against references drawn from the pool
- HPSv3: 0.30 — an open 7B preference model scoring the render
HPSv3 returns a score for how much a person would prefer an image given its description; it was trained on many human pairwise choices. The pairwise judge compares the candidate painting next to four references drawn from the pool (guided by a brief of what to weigh such as bleeds and soft edges), presents each comparison in both orders, and reports the share of comparisons the candidate wins. The only standard for that judge is the pool; thus the pool encodes the curator’s taste.
The reproducer ran three experiments differing only by the split between the two model judges:
- judge-led: pairwise judge 0.60, HPSv3 0.30 — original mix, stopped at 110 steps
- hps-led: pairwise 0.30, HPSv3 0.60 — middle mix, stopped at 110 steps
- hps-only: pairwise 0.00, HPSv3 0.90 — validation run, stopped at 60 steps
The question: how much of HPSv3’s power can be handed to a pairwise judge that measures closeness to the curated pool? More weight for the pairwise judge encodes more of the reproducer’s personal taste and can make training harder, but both judge-on runs learned.
The RL environment and p5.brush
The environment wraps everything between the model and the reward: the p5.brush library, the system prompt restricting methods, headless Chromium rendering, and a gate that rejects cheats. p5.brush (by Alejandro Campos Uribe) simulates a medium: pigment bleeds, paper texture, stroke mass and flow fields. The prompt allows only ten of the 47 methods exposed by p5.brush: scaleBrushes, noStroke, fill, noFill, fillBleed, fillTexture, beginShape, vertex, endShape and circle — other methods (lines, hatching, custom brushes) would break the watercolor aesthetic.
The gate requires the sketch to compile, call the library (not raw p5 calls), place real pigment on canvas, and not attempt to trick the scorer (for example by writing text on the canvas).
A small prompt craft addition by the reproducer — “paint each petal two or three times, a big pass first and a smaller, more opaque one inside it” — increased color richness in outputs.
The reference pool
The pool contains 178 generated paintings split by the author’s personal preferences into two tiers: love and okay. All were produced by models: GLM-5.2 (64), Kimi-K3 (57), Qwen3-Coder-Next (35) and Qwen3.5-122B-A10B (22). Each sketch started from an openly licensed iNaturalist hibiscus photo, passed three refinement iterations with a vision model, and was then hand-rated by the reproducer; 178 sketches made the final pool.
When the pairwise judge samples four references for a comparison, half are drawn from love and half from okay so an early weak policy still faces rivals it can sometimes beat. This differs from Narreddi’s original which compared only to the top tier.
Because the pool defines the standard, the model learns to imitate what the pool contains: pointing the environment to another dataset would change the reward without code edits. This reliance on a human-built pool is a core limitation.
Problems, debugging and fixes
Before anything learned there were long flat reward curves. Key debugging steps and fixes included:
- Starting from simpler control tasks (no browser, no judges) to ensure learning could occur and to tune learning rate (the initial rate was too low).
- Adjusting LoRA parameters: Qwen/Qwen3.5-35B-A3B is a mixture-of-experts and the usual target_modules list missed many projections; using all-linear reached every linear layer and enabled learning.
- Changes in TRL’s GRPOTrainer unlocked progress: learning rate raised to 5e-5, scheduler set to constant_with_warmup, scale_rewards to none, and target_modules to all-linear.
The reproducer also fixed an OpenEnv bug where a closed socket stayed cached and blocked later calls; the fix was submitted upstream. Infrastructure issues could cause rollouts to be treated as 0.0 reward (render timeouts or scorer failures), which pollutes training; the reproducer now returns None for such rollouts and excludes them from group statistics.
Training results and what improved
With the successful configuration, all three runs learned. The reproducer stopped both judge-on runs at 110 steps (a step is ~15–18 minutes) to save compute while reward was still climbing. Mean group reward over the first and final third of each run:
- hps-only (60 steps): first third 0.58 → final third 0.71, Δ +0.13
- judge-led (110 steps): 0.45 → 0.72, Δ +0.27
- hps-led (110 steps): 0.57 → 0.82, Δ +0.24
The curves align with how much the reproducer’s taste (the pairwise judge) weighs: the more the judge, the lower the start and the noisier the climb. The pairwise judge term increased in the two runs that used it: the model won more comparisons against the pool over time. No run collapsed to identical rewards (a GRPO failure mode).
What the models learned in qualitative terms:
- First, the models stopped producing very bad paintings (near-blank canvases). In hps-only, roughly 75% of the rise came from bad rollouts becoming rare. In judge runs the collapse of bad rollouts was steeper.
- With the pairwise judge active, the top-quality paintings improved more: paint coverage roughly doubled in judge-led and hps-led runs, while hps-only barely changed. The judge provides a left-beating reference to improve against.
- The explicit prompt instruction to produce 15–30 filled shapes was ignored: the actual mean number of shapes stayed ~7–9 and n_shapes barely correlated with reward.
Across each run the median paintings become similar as training advances — GRPO learns to match the pool, so a pool focused on one subject narrows the model’s variety. More diverse output requires a more diverse pool.
Costs and runtime
Rounded resource needs for finished runs reported by the reproducer:
- Trainer: 1 H200 GPU; ~18 hours for 60 steps, ~34 hours for 110 steps
- HPSv3 scorer: an a100-large Space running for the whole run
- Environment: a cpu-upgrade Space for rendering
- Pairwise judge: Inference Providers quota for Qwen/Qwen3-VL-30B-A3B-Instruct
- The pool: one-off cost of gathering openly licensed photos, generator Inference Providers quota, and the human hours to rate
A step performs eight rollouts and takes 15–18 minutes, 70–80% of which is rendering. A single render took 69–96 seconds against a 90s deadline since the Space had no GPU and Chromium rendered the WEBGL canvas in software; p5.brush bleeds and textures are heavy pixel work. A scorer can cost more than the training that uses it because HPSv3 must stay up for the whole run.
Everything runs on HF Jobs, with the environment packaged as a Docker Space and metrics captured in trackio.
What to try next
The reproducer lists untried ideas in order of perceived promise:
- Multi-step training where the model sees its render and iterates; currently training is single-turn and the model never sees the image it paints.
- Try smaller models (evidence suggests 35B is larger than needed; 4B models already wrote valid sketches in side experiments).
- SFT on the pool sources before RL, explicitly rewarding pigment, progressively hardening the judge references (easy→hard), or cautiously widening the 10-method allowlist for more visual range.
The method is not flower-specific: Alex Yango used the same mechanism for animals, and Brendan Hogan trained canvas animations against a pool of rated clips.
Final observations
The reproduction demonstrates that RL can steer a code-emitting language model toward a curated aesthetic encoded in a hand-rated reference pool. But the central bottleneck is the pool itself: 178 model-made paintings define what this trained model considers beautiful, and building a principled, diverse reward set is a fundamentally human, curatorial task.
All artifacts are published: the recipe, the reference pool and sources, the environment, the HPSv3 scorer, adapters and rollouts for hps-only, judge-led and hps-led runs, a browsable gallery of every painting, and training curves and CSV results in the repository. The original idea is Surya Narreddi’s; the p5.brush library is Alejandro Campos Uribe’s.
