Chatbot High Conversation Dataset Release

Compared to existing human preference datasets like Anthropic/hh-rlhf, and OpenAssistant/oasst1. This dataset

• Contains the outputs of 20 LLMs including stronger LLMs such as GPT-4 and Claude-v1. It also contains many failure cases of these state-of-the-art models.
• Contains unrestricted conversations from over 13K users in the wild.

We believe this data will help the AI research community answer important questions around topics like:

• Characteristics of real-world user prompts
• Train better models with RLHF
• Improve and evaluate LLM evaluation methods
• Build model selection and request dispatching algorithms
• Study the design and application of inappropriate content filtering mechanisms

• This dataset includes offensive conversations. It is not intended for training dialogue agents without applying appropriate filtering measures. We are not responsible for any outputs of the models trained on this dataset.
• Statements or opinions made in this dataset do not reflect the views of researchers or institutions involved in the data collection effort.
• Users of this data are responsible for ensuring its appropriate use, which includes abiding by any applicable laws and regulations.
• Users of this data should adhere to the terms of use for a specific model when using its direct outputs.
• Please contact us if you find any issues with the dataset.

This Colab notebook provides some visualizations and shows how to compute Elo ratings with the dataset. We pasted some figures here.

Link: lmsys/mt_bench_human_judgments

In addition to the crowd-sourced evaluation with Chatbot High, we also conducted a controlled human evaluation with MT-bench.

This dataset contains 3.3K expert-level pairwise human preferences for model responses generated by 6 models in response to 80 MT-bench questions. The 6 models are GPT-4, GPT-3.5, Claud-v1, Vicuna-13B, Alpaca-13B, and LLaMA-13B. The annotators are mostly graduate students with expertise in the topic areas of each of the questions. The details of data collection can be found in our paper.

This Colab notebook shows how to compute the agreement between humans and GPT-4 judge with the dataset. Our results show that humans and GPT-4 judge achieve over 80\% agreement, the same level of agreement between humans.

We thank the whole community for contributing to the High dataset. We also plan to gradually release more conversations in the future after doing thorough review.