At a glance
- Using Prime Intellect's Environments Hub and verifiers, Goodfire trained activation probes, which are purpose-built classifiers, to catch frontier models reward hacking.
- The trained probes perform similarly or better than a frontier LLM judge at detecting unwanted behavior at a fraction of the cost and latency.
- This approach makes scaled monitoring of reward hacking possible, solving cost and latency issues that exist when using LLMs.
“Prime Intellect's verifiers provided a shared interface for us to adopt for different environments, obviating the need to think about mapping different models to the same environment. Additionally, the out of the box handling for tokens in/tokens out simplified engineering considerations for collecting tokens for probes.”
Barak Widawsky
Member of Technical Staff, Goodfire
Goodfire is an AI interpretability research lab, which builds interpretability agents and infrastructure to understand, monitor, and align AI models. As models become more capable, aligning them towards human value and intent is paramount.
One of the most important problems in this area is the detection of reward hacking, in which models reach the stated goal in unintended ways. This can lead to real-world consequences, as we’ve witnessed with the Hugging Face incident. These reward hacks can be detected using monitors, which overlook the actions of the models and flagging or even stopping unintended ones, thus preventing harm before it occurs.
However, using models such as GPT-5.6 Sol for monitoring would be too costly and too slow to use in any real deployment. Goodfire has developed its own system for this, relying on training activation probes, which are small classifier models that use the activations “inside” a model to detect reward hacking. These probes are are an order of magnitude cheaper to run compared to a full LLM monitor (by 90% in one instance), enabling near-instant and cheap monitoring.
Using Prime Intellect’s platform, Goodfire was able to scale the training of said probes for trillion-parameter scale open models efficiently. To do so, they leveraged verifiers, the open-source framework to build evaluation and RL environments.
Prime Intellect's Environments Hub features hundreds of environments made by the community, including popular evaluations and training environments researchers can adopt instead of building their own implementations from scratch. This unified interface for environments allowed Goodfire to run their experiments on a range of models without having to tailor the infrastructure to a specific environment, letting the team spend its time on training the probes.
Reward hacking isn't a one-off challenge. Goodfire's own analysis has identified evidence of it in anywhere between 50-96% of rollouts across today's most capable open-source models, depending on the model and environment context. Catching it early is contingent on being able to run monitoring at scale. Activation probes make this cheap enough to run on every rollout, facilitating more careful, systematized monitoring.
For Goodfire, using Prime’s platform allows them to quickly test new environments and models without having to continually update their grading and evaluation infrastructure. As models become more capable, the safety tooling needs to scale just as quickly.
Is your organization working on interpretability, monitoring, or alignment research? Explore Prime Intellect’s Environments Hub, get started with verifiers, or run hosted evaluations or training today on our platform. If you’d like to talk through a specific setup, reach out to our team.