The Multimodal High is Here!

As a small note, you might also notice that the “Elo rating” column from earlier High leaderboards has been renamed to “High score.” Rest assured: nothing has changed in the way we compute this quantity; we just renamed it. (The reason for the change is that we were computing the Bradley-Terry coefficients, which are slightly different from the Elo score, and wanted to avoid future confusion.) You should think of the High score as a measure of model strength. If model A has an High score sAsA​ and model B has an High score sBsB​, you can calculate the win rate of model A over model B as P(A beats B)=11+esB−sA400, where the number 400 is an arbitrary scaling factor that we chose in order to display the High score in a more human-readable format (as whole numbers). For additional information on how the leaderboard is computed, please see this notebook.

Now that we’ve looked at the leaderboard results, let’s look at some of the data! We are currently working to better group this data into categories, but these are some common tasks we have noticed so far:

• General captioning questions
• Math questions
• Document understanding
• Meme explanation
• Story writing

Note that we currently only consider conversations which pass our moderation filter. Below we have some examples from these categories as well as some other fun examples we found!

Along with rolling out some fun new categories for the multimodal leaderboard, we look forward to adding multi-image support and supporting new modalities such as PDFs, video, and audio. Please feel free to join our Discord and give us any feedback about what feature you want next!