When and How to Use Reinforcement Learning for Agents in the NeMo/Nemotron Ecosystem

Reinforcement learning (RL) is playing a central role in aligning language models: beyond reinforcement learning with human feedback (RLHF) for AI assistants, reinforcement learning with verifiable rewards (RLVR) has emerged for reasoning and agent tasks. RL is becoming practical for enterprise needs where teams require specialized, domain-specific agents and want to convert success criteria into training signals.

Frontier work shows RL can improve general capabilities. OpenAI trained its o-series models with large-scale RL, and DeepSeek-R1 demonstrated how group relative policy optimization (GRPO) and verifiable rewards improve math, code and reasoning behavior. NVIDIA Nemotron 3 Super was post-trained using multi-environment RL across 21 NVIDIA NeMo Gym verifiers and 37 datasets, generating about 1.2 million environment rollouts.

The NeMo/Nemotron ecosystem (for example Nemotron, NVIDIA NeMo RL, NVIDIA NeMo Gym) supplies open models, post-training recipes and environment infrastructure that integrate with ecosystem tools like OpenRLHF, PrimeIntellect, SGLang, Unsloth, veRL and vLLM.

Why RL matters for agents

Organizations need specialized agents for workflows such as security triage, scientific discovery, CLI automation, customer support, data analysis and internal tool use. Customizing open models (for example Nemotron) makes this achievable: teams can trade off accuracy and speed while retaining control over data, IP and deployment.

Prompting, RAG and tools can go far, but when an agent repeatedly makes tool-call mistakes, fails in long workflows, formats outputs incorrectly, or picks the wrong strategy, you need a training signal — RL provides that. In agentic systems the reward often comes from a verifier: code or logic that scores outputs or trajectories using tests, tool execution, schema validation, simulators, reward models, LLM-as-judge review, human preference labels or other task-specific feedback.

RAG, prompting, SFT and when to use DPO, RLVR or RLHF

Don’t start by asking "which algorithm?" — start with "which behavior do I want to increase and how will I measure it?"

A short decision matrix:

• Model lacks domain facts: use RAG or data injection.
• Model ignores required format: try prompting, then SFT.
• Model should imitate examples: use SFT (LoRA or QLoRA for efficient adaptation).
• You have preferred vs rejected outputs: use DPO.
• You can algorithmically verify success: use RLVR with GRPO.
• You need nuanced human preference alignment: use RLHF or reward modeling.
• Agent fails on long-horizon workflows with multiple tool calls/state updates: use environment-based RL with trajectory-level rewards.

For verifiable tool-use and agent workflows a common path is: SFT if needed → GRPO with verifiable rewards → evaluate → inspect failures → repeat.

GRPO as a practical default for RLVR

For RLVR, GRPO (group relative policy optimization) is often a practical starting point. GRPO generates multiple completions per prompt, scores them with a verifier, and updates the model based on relative performance within the group. Compared to PPO-style RLHF, GRPO has fewer moving parts and naturally supports rule-based rewards, making it a default for many agentic RL examples.

Variants like dynamic sampling policy optimization (DAPO) add dynamic sampling and asymmetric clipping to preserve signal and exploration diversity; group sequence policy optimization (GSPO) optimizes at the sequence level instead of token level to improve stability, notably for Mixture-of-Experts models. The remainder of this guide focuses on a practical RLVR workflow using GRPO and environment-based evaluation.

The minimal RL loop

An RL training run for LLMs or agents has seven parts:

1. Policy model: the model you train.
2. Task: the input the model receives.
3. Action: the model output, tool call, code patch, command or multi-step trajectory.
4. Environment: the system that executes the action and provides feedback.
5. Verifier: the signal that scores success or produces rewards.
6. Rollouts: sampled attempts from the current model.
7. Policy update: the training step that increases the probability of better outputs.

Start with evaluation before training: run the current model on a held-out task set, inspect failures and profile the verifier or reward function. RL works best when the model can sometimes produce the right behavior but not reliably. If the reward is wrong, RL will optimize the wrong thing.

Developers commonly struggle with data, environment design, reward design and compute decisions.

Task and training data

• For SFT you need input–output demonstrations of desired behavior. For RLVR you need tasks and environment logic with verifiers and tools that can score outputs.
• Synthetic data generation expands coverage when real examples are sparse: create task variants, edge cases, tool-call scenarios and expected outputs, then filter with validators, reward models or LLM-as-judge review.
• NVIDIA NeMo Data Designer can generate structured task datasets from scratch or seed data, control relationships between fields, batch generation and validate outputs against specs. NeMo Gym can run these tasks through environments to produce scored trajectories for SFT demonstrations, preference pairs or RL rollouts.

Synthetic data is not ground truth: keep a small human-quality seed set, deduplicate aggressively, hold out eval tasks and inspect failures before using synthetic data for training.

Agentic RL needs environments, not only datasets

For single-turn tasks a static dataset may suffice. For agentic RL across single-step, multi-step and multi-turn workflows you need an environment that defines dataset, agent harness, verifier and state. Examples:

• A long-running coding agent may require many tool calls before tests pass.
• A data-analysis agent may inspect files, run queries, produce charts and validate results.
• A scientific agent may search literature, call simulators and revise hypotheses.

Environments can be simple (parser + answer checker) or complex (sandbox with tools, files, turn limits and task-specific success criteria). Evals and environments are two sides of the same system: a good eval says whether the model succeeded; a good RL environment turns that signal into training data. NeMo Gym provides a scalable, reproducible way to build environments connecting agents, models, external systems, tools and verifiers, with tutorials for single-step, multi-step, stateful, real-world and LLM-as-judge environments.

Reward and verifier design: start simple

Many RL projects overcomplicate reward design. Begin with the simplest reward that proves the loop works. For RLVR this can be binary: +1 if the output passes the verifier, 0 otherwise.

Add intermediate signals only when they measure genuine progress. For a coding agent, for example:

• +0.1 selected the right tool
• +0.2 produced valid intermediate artifact
• +0.3 passed partial test
• +1.0 completed task
• -1.0 unsafe action

Too much shaping can teach the model to maximize a checklist instead of solving the task. Good reward functions:

• measure the real task;
• are hard to game/hack;
• fail visibly when wrong.

Before training, run your reward function on 50–100 model outputs and inspect scores manually. If reward disagrees with your judgment, fix the verifier first.

Compute: budget for training and rollouts

RL cost arises from rollout and training workloads. Both depend on batch size, model size and sequence lengths. Rollout cost specifically scales with number of tool calls, conversation turns and environment steps. Inference software such as vLLM improves rollout latency while NeMo Gym improves orchestration of tool calls.

Training cost scales with dataset size and number of policy updates; training software like Megatron and NeMo Automodel improve throughput. RL frameworks such as NeMo RL build on these inference and training tools to enable an efficient loop.

GPU needs vary by workload. Small adapter-based ~1B–8B experiments can start on a single modern GPU or a small multi-GPU node. Larger models, full fine-tuning, long-context tasks, high rollout counts or multi-step agent environments require multiple GPUs. When compute is limited, first reduce model size, max tokens, generations per prompt and parallel environments.

Start small for early experiments: smaller models are excellent for debugging data, verifiers, environments and training loops. Complex tasks need more capable models before reward curves or holdout evals show meaningful improvement.

A practical first RL run: small, verifiable, inspectable

Example: build a workplace-assistant agent that generates correct JSON tool calls from natural-language requests.

Suggested path:

1. Use the NeMo RL getting-started example to pipe-clean the setup.
2. Move to the NeMo RL + NeMo Gym Workplace Assistant tutorial, which trains Nemotron Nano 9B v2 with GRPO for tool-calling across project-management workflows.
3. Start with a simple one-step tool-use task and extend to multi-step environments.

Example steps:

• Step 1: Pick one behavior to measure (e.g., produce correct JSON tool call given a natural language request).
• Step 2: Run a baseline eval on separate train/validation files; measure valid JSON rate, correct tool/command rate, correct argument rate, execution success and unsafe action rate; classify failure types.
• Step 3: Decide if SFT is needed. If the model rarely follows format, do SFT; if it sometimes succeeds but is inconsistent, use RLVR with GRPO.
• Step 4: Build a simple deterministic verifier/reward function and test it on sample outputs; fix it if scores disagree with human judgment.
• Step 5: Run a small GRPO job: example config includes model nvidia/Nemotron-Nano-9B-v2, algorithm grpo, adapter lora, num_generations_per_prompt 8, reward tool_call_verifier, eval_interval 10, held_out_eval tool_call_eval.
• Step 6: Track the right metrics: training & validation reward, success rate, invalid outputs, unsafe actions, latency and cost; confirm improvements on held-out tasks.
• Step 7: Inspect failures at every checkpoint and promote only if the tuned model beats the baseline on held-out evals without regressing on safety, latency or general capabilities.

Continuous improvement for long-running agents

Treat long-running agents like software: log real trajectories (prompts, tool calls, observations, outputs, failures, human interventions, outcomes); convert production failures into evals; bucket failure modes; choose the lightest fix first (prompt/tool change), use SFT for repeated format/domain issues, DPO for preference quality and RLVR/GRPO when success is verifiable. Train on held-out tasks, promote only when behavior improves across success rate, safety failures, cost, latency and regression tests, and version tuned checkpoints or adapters.

The loop output: production failures → evals → environments → rewards → improved model → tested deployment.

Conclusion: how to get started

The shortest route to a useful RL run is a clear task, a trustworthy verifier, a small baseline model and a held-out eval that shows whether the model actually improved. NVIDIA NeMo and Nemotron tools — Nemotron models, NeMo RL, NeMo Gym and NeMo Data Designer — provide models, recipes and environment tooling so teams can move from task definition and data generation to training, evaluation and optimization without rebuilding the entire RL stack from scratch.