Reinforcement Learning
for Agents

How LLM agents learn from their own attempts, and how to build the environments, rewards, and training systems that make it work.

Part 1

The interaction

What the agent is, what it acts on, and what a rollout records.

Introduction
Agents and environments
PoliciesHarnessesTurns and tool callsRL environmentsSandboxesBuilding environments at scale
Rollouts and traces
Compaction, subagents, and recursive agentsSelf-play and multi-agent RL

Part 2

Turning finished work into a score, and checking that the score measures the task.

Rewards
Verifiable rewards (RLVR)Reward models, judges, and rubricsReward hacking
Evaluating agents

Part 3

How scored rollouts become weight updates, and what those updates change.

Advantages and baselines
Group-relative advantages (GRPO)Cold start and difficulty filteringCredit assignment
Policy gradients
The policy-gradient loss
On-policy and off-policy RL
Importance samplingPPO clipping and trust regionsKL penalties and reference models
Entropy collapse and exploration
RL algorithms for LLMs: PPO, GRPO, and variants
What RL changes: compute, capability, and forgetting

Part 4

Generating rollouts and training on them while the model keeps changing.

RL training systems
Rollout generationGroups, batches, and packingToken-level renderingKV cache and prefix reuseTrainer–inference mismatch
Asynchronous RL
Weight broadcast and policy versions
Scaling agent RL
Inference-heavy workloads

Part 5

Learning from a teacher's tokens or a teacher's scores, and training with adapters.

Post-training methods
On-policy distillationTeachers and privileged contextLoRA and parameter-efficient RL

Reference