Preference Model Joins Fireworks' Frontier Ecosystem

11.03.26AuthorNing Cao2-minute read

When teams post-train their own models, the training environment and its graders determine what the model learns. Poor-quality or unverified RL environments can cause models to exploit gaps and learn the wrong behaviors, wasting valuable compute resources.

By joining Fireworks' Frontier Ecosystem, Preference Model provides teams with rigorous, verified RL environments built to resist reward hacking, ensuring models come out both more capable and better aligned.

What we build

Preference Model builds, verifies and researches the RL environments models train on. We built our process for frontier labs, where the quality bar is strictest, and we focus on long-horizon problems that are hard to grade:

ML engineering and research: agents that optimize inference, write GPU kernels and curate training data.
Chip design, with more verticals on the way. If your domain needs an environment that doesn't exist yet, we build it.

Underneath every environment:

Kilo, our review agent, runs on every task we deliver and hunts for ways to pass the grader without doing the work. See for example, our previous work using Kilo to audit TerminalBench 4.
Karotte, our open-source environment framework, enforces reward-function correctness by construction.
A record of how training signals fail, built from millions of evaluation runs. Each environment we build or review makes the next one harder to game.

How to work with us

Start from a verified environment. Pick an off-the-shelf environment for ML engineering, research or chip design, or have us build one for your domain.
Audit what you already have. If your team has written its own environments, evals or reward functions, Kilo reviews them for exploitable gaps before you spend compute on them.
Train your model. Run the loop yourself on the Training API, hand it to Managed Training, or work with providers like Fireworks. The Training API already gives you full control over the environment and reward, so our environments drop into the loop you write.
Check what the model learned. We review rollouts and eval results for reward hacking, so a higher score means a better model, not a better cheater.

Companies should own their intelligence. Verified training signals are where that starts.

Help us build what comes next

Preference Model is a superintelligence data research company aiming to ensure AI goes well for everyone. The biggest problem we see with current AIs is that they don't always do what we intend. Perhaps it's because they're not capable, or perhaps it's because they are insufficiently aligned. Both come down to what they were trained on: train a model on enough bad environments and you get a model that does bad things. We're addressing both of these issues in the most direct, highest leverage way available to us. This means researching how to build RL environments better, not just for the AIs of today but for the superintelligences of tomorrow.

Over the past year, we've built RL environments for several frontier labs, and we're backed by $16M in seed funding led by a16z, with participation from SignalFire, South Park Commons, Scale Angel Group, and researchers including Fei-Fei Li, Ian Goodfellow, and Julian Schrittwieser.

If you want to join us in our mission to ensure AI keeps working in humanity's best interests, we'd love to hear from you: email hello@preferencemodel.com, or see our open roles.