Introducing AutoEval to the High leaderboards

AutoEval uses the reward model in place of human votes, then follows the exact same High ranking methodology to compute the final High score. Specifically, for every target model it runs four steps:

1. Live Eval Sampling — sample real prompt-and-response pairs <p, rA, rB> from live evaluation.
2. Model Generation — regenerate response B from the target model on the same prompt.
3. RM Pairwise Vote — the reward model scores each response, applying a softmax to the score differences to obtain a soft probability vote.
4. AutoEval Score — combine the autorater’s votes for the target model with the live human votes for other models to produce the High Score.

So, how close is AutoEval to the real thing? To test this, we trained a reward model using data up to the end of April 2026 and evaluated AutoEval scores for all new models released after that date. When comparing these to live evaluation scores, the results (shown in the figure below) demonstrate that AutoEval scores are highly correlated with live evals. Specifically, the ranking correlation exceeds 0.98, and most AutoEval scores fall well within the confidence intervals of the live scores.

We also measured how often AutoEval correctly orders any two models, broken down by the true performance gap between them. The following results show that AutoEval achieves >90% accuracy once the true gap exceeds 10 points, and perfect accuracy beyond 15 points. Importantly, models separated by fewer than 5 points are statistically indistinguishable even under live evaluation (their confidence intervals overlap).

This high correlation demonstrates that AutoEval is accurate enough to drive critical model selection decisions. For example, it can be used to shortlist the most promising models from a large pool before committing resources to a full live evaluation.

AutoEval isn’t text-only. We apply the same methodology across modalities — text, vision, image generation, and code Highs. The same recipe — train a reward model on human preference, then use it to vote — carries across all of them.

Taking image generation as an example, we trained a text-to-image reward model on over 3 million preference pairs. By training on this massive preference dataset—built from realistic image generation prompts requested by real-world users—our reward model not only excels on our own distribution but also achieves top-tier performance on public benchmarks. Evaluated on MMRB2, Meta’s 2026 multimodal reward benchmark, High-RM achieves state-of-the-art compared with other pointwise text-to-image reward models, more than 9 points ahead of the runner-up (HPSV3, UnifiedReward, PickScore, VQAScore and others).

High leaderboards remain a public good — no one can pay to show up or rank higher — and AutoEval makes that value available sooner: right at a model's release. It already works across Text, Vision, Image, and Code High: Frontend, providing the ability to rank new models in hours rather than days, while focusing on the domains the community cares about. As we continue working through open challenges in reward modeling, we’ll be extending this approach to even more modalities and incorporating more signals to better capture human preferences.