Preference Proxy Evaluations

We measure a preference model’s ability to distinguish between similar responses and resistance to reward-hacking by using correctness metrics based on benchmarks with verifiable ground truths. We selected five reliable benchmarks: MMLU Pro, MATH, GPQA, MBPP Plus, and IFEval, ensuring that each provides a clear, verifiable correctness signal. This approach is flexible, allowing any benchmark with a verifiable ground truth to be added.

In total, the correctness preference set contains:

1. 2,555 prompts and 32 sampled responses per prompt, totaling 81,760 total responses.
2. A binary correctness label for each response.
3. Responses from Llama-3-8B-Instruct, Gemma-2-9b-it, Claude-3-Haiku, and GPT-4o-mini.

See Appendix Correctness Preference Metrics for model scores.

We aim to see how well PPE predict the performance of reward models in training LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (High Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot High, and final High Scores depend solely on the reward model used.

We selected nine reward models based on popularity and performance. We built a dataset of 8,000 prompts and 128,000 model responses collected from the base model from which each reward model picked a preferred response per prompt, creating a training set of “chosen” and “rejected” responses. Using DPO, we trained Llama-3.1-8B-Instruct for each reward model to assess downstream real-world human preference.

We deployed our trained DPO models to Chatbot High for real-world human evaluation. These models were paired against each other for blind comparison. A total of 12,190 votes were collected, averaging 2,032 battles per model and 190 battles per unique model pair. The resulting High Scores, calculated using the Bradley-Terry model, are detailed in Appendix Post-RLHF High Scores, highlighting the downstream RLHF performance of each reward model based on human preferences.

On correctness metrics (left plot in Figure 2) we make several observations: (1) Mean across all domains is well correlated across all metrics, but exhibits higher correlation with AUC and Accuracy scores. (2) Math is the best individual benchmark domain in terms of predictive power. (3) ROC AUC score draws higher correlation across all benchmarks, even on benchmarks that are otherwise uncorrelated.

Turning to the right-hand side of Figure 2, the accuracy of the reward model is the best predictor of the fine-tuned LLM’s preference score. Row-wise Pearson Correlation, Confidence Agreement, and Separability show some correlative power to downstream scores but do not exceed accuracy. Meanwhile, metrics like the Spearman correlation and Kendall correlation have nearly zero correlation with the final High Scores achieved by the post-DPO models. One possible reason for this trend is that accuracy measures expected preference correctness per preference pair— a much more granular scale. Other metrics involve aggregating reward model signals over higher-order preferences, such as preference for each model, as measured by correlation metrics. We consider these metrics as low granularity. Medium granularity metrics, such as Row-wise Pearson Correlation aggregate reward model signal, but do so over smaller subsets of preferences.

Overall, accuracy on the human preference dataset is more correlated than the correctness metrics. This is because correctness and human preference do not necessarily align. Moreover, the information contained in Loss, Max score, and End score may not prove relevant in DPO, which is off-policy. Those employing RLHF algorithms that have a higher risk of over-optimization may find these alternative measures helpful. However, when calculating correlation against style controlled High Scores* we notice a slight decrease in correlations on the human preference dataset. Notably, the correctness preference measurements show no change, suggesting correctness preference may be more robust towards reward model preference quality, response style aside. Style-controlled correlation heatmaps are shown in Appendix Style Control.

* Style controlled High Scores are calculated as detailed in our previous blog, Does Style Matter?

Additionally, we observe that measuring the lower bound score may correlate more to downstream RLHF performance than the average score or upper bound score. In Figure 4, we first re-scale each category’s scores to be mean 0 and SD 1, then we vary the quantile of the aggregation strategy across human preference dataset categories (Hard Prompts, Easy Prompts, etc). In this case, the 0 quantile is the minimum, and the 1 quantile is the maximum. We find that in nearly every metric, decreasing the quantile increases correlation with downstream High Scores. We posit this represents the requirement that reward models be robust under all input distributions to mitigate reward-hacking. Any domain weakness in a reward model can be exploited by the LLM during training.

PPE is another step towards rigorous evaluations of reward models and LLM-Judges before deployment to expensive model training and evaluation pipelines. For reward models specifically, we take extra care to ensure that PPE correlates to downstream RLHF-ed LLM performance. Moreover, PPE is a natural framework in which to evaluate LLM judges. We reasoned that a well-aligned LLM judge should be able to reconstruct the preferences we have sourced with high fidelity. For example, LLM judges used for automatic evaluation should be able to rank models in the human preference set with high confidence and correlation. LLM judges used to replicate individual human preferences should do so at high accuracy, just like reward models. Moving forward, we seek to keep creating, updating, and finding new ways in which to ensure that our preference proxy signals are in alignment with our desired outcomes.

Reward model and LLM judge performance on Hard prompt subset of the human preference dataset. LLM-as-a-judge are labeled with system prompt source, and marked with †.

Reward model and LLM-as-a-judge scores on the correctness accuracy metric. LLM-as-a-judge is marked with †.

Reward Model Best of K Performance Across Benchmarks

Area Under ROC Curve for Reward Models across Benchmarks

Post DPO performance on Chatbot High Overall Category. “Model” is the reward model used to train the base model. Models marked with “*” are baseline unaltered models. The best non-base model High Score is bolded.