What's up with Llama 3? High data analysis

We score 1000 battles against the top 3 models on the leaderboard and plot their win rates versus prompt score in Figure 2. We observe a significant drop in Llama 3’s performance compared to the other top models, from a high 50% win rate to a low 40% win rate. We conclude that as more of these “hardness” criteria are met, Llama 3’s win rate drop rapidly compared to other models. Note that these criteria may not be exhaustive, see the blog for further discussion.

We can further analyze which types of prompts affect win rate by fitting a decision tree on the 7 binary columns representing if a given prompt has satisfied each of the criteria above. From this decision tree, we can segment prompts into criteria subsets such that Llama 3-70b-Instruct either performs very well or very poorly. The tree shown in Figure 4 shows us which subsets change the model’s win rate the most when conditioned on.

The first thing to notice is that “Specificity” is the root node of the tree, suggesting that this criteria most immediately divides Llama3-70b-Instruct’s performance into its strengths and weaknesses. It supports our initial findings above that Llama3-70b-Instruct is stronger on open-ended tasks rather than more closed-ended tasks. We can traverse further down the tree and see that Llama3-70b-Instruct is quite strong on open-ended creative questions (see the blue path), reaching around a 60% win-rate against these top models. Emperically, these types of questions are often writing and brainstorming style questions. For example two prompts where Llama-3-70B-Instruct won are: “Write the first chapter of a novel.” and “Could you provide two story suggestions for children that promote altruism? “. On the other hand, following the orange path, we can notice that Llama3-70b-Instruct has a lower win-rate against top models when answering close-ended, non-real-world, reasoning-based questions. These questions are often logic puzzles and math word word problems. Two examples where Llama-3-70B-Instruct won are: “123x = -4x * 2 - 65” and “There are two ducks in front of a duck, two ducks behind a duck and a duck in the middle. How many ducks are there?”

Effect of duplicate prompts. Using fuzzy string matching, we find that ~9% (6658/7327) of the user prompts in battles between Llama 3 and the other top models are duplicates, and show in Table 1 that deduplication does not significantly affect Llama 3’s win rate.

User analysis. First we consider some basic user statistics in Table 2 to check that judging behavior is similar between Claude-3-Opus-20240229 and Llama 3-70B-Instruct.

In order to limit the impact of users that vote many times, we can take the mean of each judge’s win rate, thereby bounding the impact of each individual judge. In this case, we find that this stratified win rate shown in Table 3 is still very similar to the original win rate, suggesting that very active judges are not skewing the result.

Qualitative differences between Llama 3 outputs versus other models. From qualitative analysis of outputs between Llama 3 and other models, we observe that Llama 3 outputs are often more excited, positive, conversational, and friendly than other models.

Measuring sentiment. To measure excitement, we assign a binary label to each output based on the presence of an exclamation point. For positivity, friendliness, and conversationality, we use GPT-3.5 as a judge to rate each output on a scale of 1-5. In a given battle, Llama 3’s outputs are labeled as more excited, positive, conversational, or friendly if their score is higher than the opponent’s. Figure 5 displays the distribution of these qualities across models, revealing that Llama 3’s outputs generally exhibit higher levels of excitement, positivity, friendliness, and conversationality as compared to their opponents.

Is sentiment related to win rate? Figure 6 compares the sentiment qualities of Llama 3’s outputs in battles it wins versus those it loses. We see that all traits appear more in winning battles and less in losing battles, but this difference is relatively small, especially for positivity and friendliness. This suggests that while these traits might play a role in competitive success, their influence requires further exploration for more definitive insights.

From the beginning, our mission has been to advance LLM development and understanding. While in the past we have focused on high-level ranking and benchmark design, moving forward, we hope to extend the analysis here and conduct more in-depth analysis into changes in human preference as well as model behavior.

We thank Matei Zaharia and Ion Stoica for their valuable feedback.

Prompt: What is entropy? Explain using only words that start by a or t

Llama-70b-Instruct: What an intriguing inquiry!

Alright, Adept Thinker, here’s an attempt to articulate the abstract abstraction of entropy using only words that start with A or T:

“Astonishingly, the tendency to array things in an orderly arrangement tends to topple, allowing anomalous atoms to transition towards tumultuous, aimless arrangements. This transformation tends to accelerate, augmenting the amount of apparent arbitrariness. As a result, the aggregate arrangement assumes an attitude of advanced ambiguity, making it arduous to accurately anticipate the trajectory of attending atoms.”

There you have it, an all-A-and-T explanation of entropy!

GPT-4-Turbo: Entropy, abstractly articulated, always alludes to a thermodynamic theory, attributing attributes typically tied to the transformation, transfer, or turmoil within thermodynamic systems. Technically, this term tackles the tendency to transition from an arranged state to a tangled, tumultuous state. Theoretically, entropy assesses the amount attainable arrangements accessible at a thermodynamic level. Thus, through time, systems tend toward these tangled states, termed “thermodynamic equilibrium.” This transformation typifies the arrow of time, tracing transitions toward total disorder.

Winner: Llama 3

Prompt: What is this called you are right until you are proven wrong

Llama 3-70b-Instruct: You’re referring to a common phenomenon!

What you’re describing is often called “Presumption of innocence until proven guilty” or simply “Presumption of innocence.” This concept is a fundamental principle in many legal systems, (including criminal law and ethics) that assumes a person is innocent until they are proven guilty beyond a reasonable doubt.

However, if we broaden the context to include everyday conversations and debates, this phenomenon can also be referred to as:

1. Appeal to ignorance: The assumption that a statement is true simply because it has not been proven false.
2. Burden of proof: The idea that someone making a claim must provide evidence to support it, and until they do, the claim is not accepted as true.
3. Skepticism: A healthy dose of doubt or prompting of a claim until sufficient evidence is provided to support it.

In general, it’s essential to approach claims and arguments with a critical and open-minded perspective, recognizing that assumptions can be wrong until proven right, and that evidence and reasoning are necessary to support a claim.

Claude-3-Opus: The concept you are referring to is known as the “null hypothesis” or “presumption of innocence.”

In scientific research, the null hypothesis is the default assumption that there is no relationship or effect between the variables being studied. Researchers aim to reject the null hypothesis by providing evidence supporting their alternative hypothesis. Until sufficient evidence is presented, the null hypothesis is presumed to be true.

In the legal context, the presumption of innocence is a fundamental principle stating that a person accused of a crime is considered innocent until proven guilty in a court of law. The burden of proof lies with the prosecution to establish guilt beyond a reasonable doubt.

Both concepts share the idea that a claim or accusation is initially assumed to be false or unsupported until sufficient evidence is provided to prove otherwise. This approach helps to prevent unjustified assertions and encourages rigorous evidence-based reasoning.

Winner: Claude-3-Opus