A skill-based, agent-led autoresearch workflow was demonstrated for reinforcement learning that automates environment setup, experiment execution, monitoring, metric analysis, and result summarization. The stack used NVIDIA NeMo RL and NeMo Gym, an NVIDIA Brev instance with an L40S 48 GB GPU, and a frontier coding agent (Codex with GPT 5.5). The design goal is to offload repetitive engineering and iteration to agents while keeping human researchers responsible for objectives, milestone review, and strategic decisions.
Why this matters for RL research
Meaningful RL metrics typically require a working experimental infrastructure (environments, data generation, checkpointing, logging) before they appear. Autoresearch—an open source Python approach popularized by Andrej Karpathy—lets autonomous agents convert high-level goals into hypotheses, edit and test real codebases, and return validated results to humans, enabling faster iteration while retaining control over data, IP, and deployment.
Tools and components used
- Codex with GPT 5.5 for code navigation, reasoning, and tool use.
- NVIDIA NeMo RL and NVIDIA NeMo Gym (parts of the NVIDIA NeMo framework) for training and live-generated environments.
- NVIDIA Brev remote instance with one NVIDIA L40S 48 GB GPU.
- Qwen family models: Qwen3-VL-2B-Instruct and Qwen3-1.7B in examples.
- Agent skills used: Brev-etiquette (instance hygiene and artifact placement), session-memory (durable session diary), and autoresearch (experiment loop, ledger, stop rules).
Demonstrated agent capabilities
The workflow shows three main capabilities:
- Full-stack autonomy: installing and configuring software, resolving dependencies, managing GPU memory and disk, handling checkpoints, launching and debugging runs.
- Goal-driven autoresearch: profiling a baseline, proposing and testing hypotheses, running experiments, and iterating toward a research objective.
- Paper-to-code: reading a paper, forming an implementation plan, translating algorithms into code, adding tests, and starting validation training.
Example: star_count environment and SFT campaign
In the examples, Codex first set up a NeMo RL + NeMo Gym stack and ran a smoke test with Qwen3-VL-2B-Instruct. It subsequently designed and implemented a new NeMo Gym visual counting environment called "star count", which generates images with colored stars at varying resolutions and asks the model to count stars by color.
Key results reported by the agent for the star_count SFT campaign:
- Baseline (64-example held-out): 25.0% exact accuracy, mean reward 0.705208.
- Initial LoRA SFT smoke run: 78.125% exact accuracy.
- Scaled runs: 2048-train / 256-val, 160 steps → 95.3125% exact accuracy.
- Best run: 4096-train / 512-val, 320 steps → 96.875% exact accuracy, mean reward 0.992188; best checkpoint was step_240.
- Overall improvement: from 25.0% to 96.875% exact accuracy (71.875 percentage points). Remaining errors were near-miss off-by-one counting mistakes by color, suggesting targeted hard-case augmentation as the next experiment.
A typical stack validation and smoke test in this workflow took roughly 40 minutes to an hour.
Paper-to-code example: implementing OAPL
The workflow also demonstrated paper-to-code for the paper "LLMs Can Learn to Reason via Off-Policy RL" (arXiv:2602.19362), which introduces Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL), a long-lag off-policy method. The agent read the PDF, extracted the algorithm, inspected NeMo RL hook points, produced an implementation plan, translated the algorithm into code, added unit tests and documentation, and launched a validation campaign training Qwen3-1.7B on the DAPO-math-17K dataset with a 10-hour budget.
Reported behavior in validation runs included higher accuracy for OAPL with fewer training steps compared with a GRPO variant (DAPO), while OAPL consumed significantly more tokens—raising follow-up research questions about verbosity vs. efficiency.
Practical notes, failure modes and best practices
Observed failure modes and mitigations:
- Context drift: long sessions and context compaction can cause the agent to lose the objective or stop rules; persistent session-memory helps.
- Local housekeeping: without explicit Brev-etiquette, logs, checkpoints and caches scatter across the filesystem.
- Steering drift: mid-run interactions can derail an auto session; keep steering questions focused.
- Low-signal loops: short runs or tiny batches give poor training signals; human oversight should detect and correct this.
Suggested practices:
- Treat agent skills as evolving workflow instructions and update them with institutional knowledge.
- Keep human judgment central for difficult research questions; use staged approvals for implementations and long runs.
- Perform planning and brainstorming before launching long-running sessions.
- Define explicit budgets (time, GPU-hours, max experiments) to constrain the agent and focus on high-value hypotheses.
Conclusion
Agent-led autoresearch using NeMo RL, NeMo Gym, Brev, Codex and a small set of operating skills can significantly reduce setup, debugging, and repetitive experiment effort in RL research. The approach enables rapid environment creation, baseline profiling, iterative experimentation, and even paper-to-code workflows, while preserving human control over goals and final decisions. The demonstrated numeric outcomes—such as improving Qwen3-VL-2B-Instruct exact accuracy from 25.0% to 96.875% on the star_count task—illustrate the potential speed-up in experiment iteration.
Getting started
- Try the NeMo RL Autoresearch Launchable on Brev.
- Review NVIDIA-verified Agent Skills documentation and the NVIDIA/skills GitHub repository (including the NeMo RL Auto Research skill).
- Explore NVIDIA NeMo RL and NVIDIA NeMo Gym for RL training and live-generated environments.
(Descriptions and numbers in this article are drawn from the reported agent runs, logs and summary outputs presented in the demonstration.)
