A watercolor coding model cut sub-0.3 rollouts from 99 to 16. Its style narrowed too
The best result in a new experiment that taught a coding model to paint watercolor flowers is a count, not an image. One training run produced 99 rollouts scoring below 0.3 in its first third and 16 in its final third.
Its paintings also became more alike. The reward came from a hand-rated pool built around one flower, so the model learned to repeat the curator's preferences. A higher score and a narrower style were two results of the same objective.
Sergio Paniego published the environment, trainer, models, reference pool, metrics, and every rollout. That record lets readers inspect what aesthetic reinforcement learning measured rather than judge a highlight reel.
Painting is an executable agent task
The open reproduction, based on an idea by Surya Narreddi, uses Qwen3.5-35B-A3B. Instead of generating pixels, the model writes JavaScript for p5.brush, a p5.js library that simulates pigment bleed, paper texture, and brush mass. Headless Chromium renders each sketch at 600 by 600 pixels. An OpenEnv environment then scores the picture.
Only 10 of p5.brush's 47 methods appear in the system prompt. The model may create filled shapes and circles, add vertices, and control texture or bleed. A gate rejects code that fails to compile, bypasses the library, leaves the canvas effectively blank, or draws text to trick the scorer.
Code is the action, a browser is the environment, and the image receives the reward. The recipe and training code are public.
The reference pool defines good
Three LoRA adapters started from the same base model. Every step sampled eight paintings of a peach hibiscus. Small rewards covered honest rendering and code length. Two visual judges supplied most of the score.
HPSv3 is a 7 billion parameter preference model trained from human image choices. A pairwise judge, Qwen3-VL-30B-A3B-Instruct, compared each new painting with four references. The reward mixes were:
| Run | Pairwise judge | HPSv3 | Steps |
|---|---|---|---|
hps-only | 0.00 | 0.90 | 60 |
judge-led | 0.60 | 0.30 | 110 |
hps-led | 0.30 | 0.60 | 110 |
The references came from a pool of 178 generated paintings that Paniego sorted into love and okay tiers. Four open models made the candidates. Some generation rounds used openly licensed iNaturalist photographs as guides, but the pool contains no human-made painting.
The pairwise judge measured resemblance to a model-generated set filtered by one person. HPSv3 supplied a broader preference signal. Their weights decided which preferences training would optimize.
Rising reward mostly removed the bottom
Mean judge-led reward increased from 0.45 in the first third to 0.72 in the final third, while sub-0.3 rollouts fell from 99 to 16. For hps-led, mean reward rose from 0.57 to 0.82 and the weak-output count fell from 37 to 4.
The judge-led rollout dataset includes 861 paintings with their JavaScript, step numbers, component rewards, and code statistics. Paint coverage doubled from 0.11 to 0.23, and the pairwise judge term climbed from 0.36 to 0.70.
The simpler hps-only run produced a different pattern. Roughly three quarters of its group-mean gain came from weak paintings becoming rare. The best image in each group barely improved. Once HPSv3 recognized a flower with petals and a stem, it applied little pressure for denser or more expressive paint.
An instruction in the prompt requested 15 to 30 filled shapes. The 470 hps-only rollouts averaged 7.6, and shape count had effectively zero correlation with reward. The policy followed the score instead of the sentence.
Optimization narrowed the style
The pairwise judge moved the best outputs and increased paint coverage, but later paintings within each run often looked like versions of one hibiscus. The one-subject pool rewarded that repetition.
Supporting more variety would mean curating a broader pool. That requires more source photographs, candidate generation, and human rating. The project estimates that the pool is most of the work when adapting the method to another domain.
In this pipeline, a person selects references, labels preferred examples, chooses judges, and sets their weights. Training turns those editorial choices into gradients.
Several limitations complicate the curves. The longer runs stopped at step 110 while reward was still increasing. p5.brush is random, so rerunning the same JavaScript can produce a different image. Each step receives a different draw from the reference pool, which makes comparisons within a step valid but muddies comparisons between steps. Nineteen of the 110 judge-led groups have seven rollouts instead of eight because failed renders or scorer responses were excluded.
An audit for creative rewards
Save every rollout, including the ugly ones, and report the bottom, median, and top separately. Fewer failures and better best-case work are different outcomes.
Preference needs a diversity measure beside it. Teams can track repeated composition, palette, geometry, and subject coverage, then inspect random samples. Aesthetic averages cannot reveal polished repetition by themselves.
Give each reward component its own log. Service failures belong in an infrastructure metric, not the art score. This project changed render timeouts and missing scorer responses from 0.0 to None after finding that operational faults were training the model as if they were bad paintings.
The reference pool should be public. Otherwise, nobody outside the project can tell whose preferences became the objective. Our GRPO schema-compliance article found a comparable pattern in a less subjective task: the behavior named by the reward improved most. The BenchMIRT analysis explains why the name of a score can imply more than its measurement supports.
This model learned to suppress weak outputs and beat a selected pool. Publishing that pool is what allows a viewer to decide whether the learned taste deserves the higher score.
