Multi-Agent Systems in PRIME-RL
Multi-Agent Systems in PRIME-RL
Today, the Prime Intellect RL stack expands from training individual agents to multi-agent systems. You can now program arbitrary interactions between agents, choose which roles learn, and assign credit across the complete interaction.
Two recently introduced abstractions set the stage for this development: verifiers v1 introduced the primitives for running a single agent on a programmable task in a swappable harness and runtime, and prime-rl gained an algorithms layer to make the mapping from rollouts to training signal programmable.
Today, we are bringing both pieces together by introducing abstractions that enable multi-agent training and evaluations first-class. Some interesting examples that can now be expressed:
- Agentic Judging — solver traces are graded by a judge
- Self-Play — a model playing against itself
- User-Sim — a user agent interacting with an assistant
This post introduces the two main abstractions, the Agent and Env, explains the design choices behind them and tours several interaction patterns that we believe open promising new directions for agentic RL.
Agent
verifiers v1 natively supported all the primitives that define a single agent rollout:
Tasksetsupplies a set ofTaskobjectsHarnessdefines the program that drives the modelRuntimedefines where the program executes
An Agent just becomes the natural home for each of these. The core signature is Agent.run(task: Task) -> Trace — when given a Task the agent produces a Trace, the artifact produced by the rollout.
class Agent:
async def run(self, task: Task, *, runtime: Runtime) -> Trace:
"""Produce a trace given a task."""
...
Behind the scenes, the agent manages the lifecycle of all the hidden machinery which enables you to run any agent in any runtime on any task. By making the abstraction first-class, you can easily script over it. Multi-agent environments are then just one instantiation of a program over agents.
Env
The Env now becomes the home for multi-agent training and evaluation. Its core signature is Env.run(task: Task, agents: Agents) -> None — it is passed an initial task and a list of pre-initialized agents and then programs the full multi-agent control flow; every finished agent run automatically joins the resulting Episode.
class Env(ABC):
@abstractmethod
async def run(self, task: Task, agents: Agents) -> None:
"""Run a single multi-agent episode."""
...
Anything that could be expressed in verifiers v1 collapses to a one-line run method in SingleAgentEnv.
class SingleAgentEnvConfig(EnvConfig):
agent: AgentConfig = AgentConfig()
class SingleAgentEnv(Env[SingleAgentEnvConfig]):
async def run(self, task: Task, agents: Agents) -> None:
await agents.agent.run(task)
We will go through four already implemented multi-agent envs; we think they are both useful in their own right and instructive demonstrations of what these abstractions make possible.
Agentic Judging
Our post on scaling agentic RL ended with a problem that deterministic grading cannot solve: fixed graders are often too narrow. For example, in software engineering, tests may assert a particular implementation detail, making a valid solution incorrectly receive zero reward.
A naive LLM judge would not solve this problem — bounded by a single call, it would be unreasonable to expect an accurate verdict. An agentic judge, on the other hand, is capable of exploring the codebase, looking at the failing tests and overruling the deterministic tests.
The AgenticJudgeEnv defines the sequential interaction between a solver and judge agent.
class AgenticJudgeEnvConfig(vf.EnvConfig):
solver: vf.AgentConfig = vf.AgentConfig()
judge: vf.AgentConfig = vf.AgentConfig()
class AgenticJudgeEnv(vf.Env[AgenticJudgeEnvConfig]):
async def run(self, task, agents) -> None:
solution = await agents.solver.run(task)
if not solution.ok:
raise
await agents.judge.run(JudgeTask.from_trace(solution))
Illustrative example. Check out the full implementation here.
Self-Play
A central bottleneck in LLM RL is task scarcity. Useful learning signal requires tasks that are close to the agent's current capabilities. Self-play offers an alternative to static tasksets by allowing the model to help generate its own curriculum. As the model improves, it can create harder tasks, be a stronger opponent, or find new failure cases for itself. We believe that self-play will become an important technique to push the frontier.
In this section, we introduce two variants of self-play — Proposer-Solver and Kuhn-Poker.
Proposer-Solver
In ProposerSolverEnv the proposer receives a seed topic and constructs a new task, which is then solved by a group of solvers. Each solver is rewarded for answering correctly while the proposer is rewarded for calibration to the solver population. If every solver succeeds, the task was too easy; if none succeed, it may be too difficult or invalid. The environment's learnability peaks at a 50% solve rate, encouraging the proposer to generate tasks that yield the maximum training signal for RL (inspired by Zhao et al. (2025), Absolute Zero).
The interaction also requires role-aware credit assignment. Solver attempts should be compared with attempts on the same proposed problem, while proposer traces should be compared with other proposals generated from the same seed task. Classic GRPO cannot represent this hierarchy, so we implemented Hierarchical GRPO to preserve these comparison sets without mixing roles or problem difficulties.
class ProposerSolverEnvConfig(vf.EnvConfig):
proposer: vf.AgentConfig = vf.AgentConfig()
solver: vf.AgentConfig = vf.AgentConfig()
n: int = Field(4, ge=1)
class ProposerSolverEnv(vf.Env[ProposerSolverEnvConfig]):
async def run(self, task: vf.Task, agents: vf.Agents) -> None:
proposed = await agents.proposer.run(task)
solve_task = SolveTask.from_trace(proposed)
async with asyncio.TaskGroup() as tg:
for _ in range(self.config.n):
tg.create_task(agents.solver.run(solve_task))
async def finalize(self, task: vf.Task, episode: vf.Episode) -> None:
rate = solve_rate(episode.traces) # mean solve rate
for trace in episode.traces:
if trace.agent.name == "proposer":
trace.record_metric("solve_rate", rate)
trace.record_reward("learnability", 4.0 * rate * (1.0 - rate))
Illustrative example. Check out the full implementation here.
Kuhn-Poker
In KuhnPokerEnv two models play Poker. The environment maintains the private cards, public game state, and legal actions. As the policy improves, it also becomes a stronger opponent, producing a moving curriculum without a separate opponent service.
Because different roles can have structurally different reward distributions, our training system supports Role-Conditioned Advantage Estimation (RAE) for this setting. Rather than measuring every agent against one shared baseline, RAE measures each role relative to its own reward history.
User-Sim
Turn-level interleaving is useful beyond turn-based games. Many assistant tasks cannot be represented by a single prompt and response: the user has private context, reveals information over time, reacts to the assistant, and decides when the goal has been met.
The UserSimEnv models users as agents, and the episode is a turn-by-turn conversation between the user and assistant agents.
Both Traces remain visible in the Episode. The simulated user is frozen by default, while the assistant's Trace is scored against the original Task and used for training.
This makes user simulation another composition of the same primitives rather than a separate evaluation path. Different user populations, personae, hidden goals, and interaction policies can be plugged into the same interface, while the assistant can be configured differently.
class UserSimEnvConfig(vf.EnvConfig):
assistant: vf.AgentConfig = vf.AgentConfig()
user: vf.AgentConfig = vf.AgentConfig()
class UserSimEnv(vf.Env[UserSimEnvConfig]):
async def run(self, task, agents):
user_task, assistant_task = ..., ...
async with (
agents.user.interaction(user_task) as user,
agents.assistant.interaction(assistant_task) as assistant,
):
ask = await user.turn("Hello! How can I help you today?")
while True:
answer = await assistant.turn(ask.last_reply)
if answer.terminated:
break
ask = await user.turn(answer.last_reply)
if ask.terminated:
break
Illustrative example. Check out the full implementation here.
Agents beyond RL
We expect the agent abstraction to be useful beyond training and evaluation. We are already building powerful synthetic data generation and curation pipelines. Each agent's trace is unified and auditable data artifacts.
To illustrate, here is Laguna-S2.1 in the pool harness finding the latest released verifiers version.
import asyncio
import verifiers.v1 as vf
async def main():
agent = vf.make_agent(
vf.AgentConfig(
model="poolside/laguna-s-2.1",
harness={"id": "pool"},
runtime=vf.PrimeConfig()
)
)
task = vf.Task(
vf.TaskData(
prompt="Find the latest released version of the `verifiers` Python "
"package on PyPI. Answer with just the version string."
)
)
async with agent:
trace = await agent.run(task)
print(trace.last_reply)
asyncio.run(main())
Multi-agent support releases today in verifiers 0.3.0 and prime-rl 0.8.0 — we are excited about empowering researchers to express multi-agent RL and push the frontier for open-source AGI.
Citation
@article{primeintellect2026multiagent,
author = {Konstantin Dunas and Mika Senghaas and Eli Gottlieb and Prime Intellect Team},
title = {Multi-Agent Systems in PRIME-RL},
journal = {Prime Intellect Blog},
year = {2026},
month = {August},
note = {https://www.primeintellect.ai/blog/multi-agent-systems}
}