MMLU
Multiple-choice questions across 57 subjects. Saturated at the frontier.
Agent Index weight
0%
shown for context only
Models tested
116
from the benchmark’s leaderboard and reports
Results
| # | Model | Score | Normalised | Harness | Cost/task | Date |
|---|---|---|---|---|---|---|
| 1 | GPT-4o (2024-08-06) Supported OpenAI | 88.1% | source | — | — | |
| 2 | Claude-Sonnet-3.5 Estimated Anthropic | 87.3% | source | — | — | |
| 3 | DeepSeek-V3 DeepSeek | 87.2% | source | — | — | |
| 4 | Gemini 1.5 Pro (Feb 2024) Google | 86.9% | source | — | — | |
| 5 | GPT-4 OpenAI | 86.4% | source | — | — | |
| 6 | Llama 3.3 70B Meta | 86.3% | source | — | — | |
| 7 | Qwen2.5-72B Estimated Alibaba (Qwen) | 85.3% | source | — | — | |
| 8 | Phi 4 Microsoft | 84.8% | source | — | — | |
| 9 | Claude 3 Opus Estimated Anthropic | 84.6% | source | — | — | |
| 10 | Llama 3.1-405B Meta | 84.5% | source | — | — | |
| 11 | Qwen2-72B Alibaba (Qwen) | 82.4% | source | — | — | |
| 12 | Amazon Nova Pro Amazon | 82.0% | source | — | — | |
| 13 | GPT-4o mini OpenAI | 81.8% | source | — | — | |
| 14 | GPT-4 Turbo OpenAI | 81.3% | source | — | — | |
| 15 | Llama 3.2 90B Meta | 80.3% | source | — | — | |
| 16 | Llama 3.1-70B Meta | 80.1% | source | — | — | |
| 17 | Mistral Large 2 (Nov 2024) Mistral | 80.0% | source | — | — | |
| 18 | Qwen2.5-14B Alibaba (Qwen) | 79.9% | source | — | — | |
| 19 | Gemini-2.0-Flash Google | 79.7% | source | — | — | |
| 20 | Llama 3-70B Meta | 79.3% | source | — | — | |
| 21 | Yi-Large 01.AI | 79.3% | source | — | — | |
| 22 | Qwen2.5-Coder-32B Alibaba (Qwen) | 79.1% | source | — | — | |
| 23 | Claude 2 Anthropic | 78.5% | source | — | — | |
| 24 | DeepSeek-V2 (MoE-236B, May 2024) DeepSeek | 78.4% | source | — | — | |
| 25 | phi-3-medium 14B Microsoft | 78.0% | source | — | — | |
| 26 | Gemini 1.5 Flash (Sep 2024) Google | 77.9% | source | — | — | |
| 27 | Mixtral 8x22B Mistral | 77.8% | source | — | — | |
| 28 | Amazon Nova Lite Amazon | 77.0% | source | — | — | |
| 29 | Claude 1.3 Anthropic | 77.0% | source | — | — | |
| 30 | Yi-34B 01.AI | 76.3% | source | — | — | |
| 31 | Claude 3 Sonnet Anthropic | 75.9% | source | — | — | |
| 32 | Gemma 2 27B Google | 75.7% | source | — | — | |
| 33 | phi-3-small 7.4B Microsoft | 75.7% | source | — | — | |
| 34 | Qwen2.5-Coder-14B Unknown | 75.2% | source | — | — | |
| 35 | Qwen1.5-32B Alibaba (Qwen) | 74.4% | source | — | — | |
| 36 | Claude 3.5 Haiku Anthropic | 74.3% | source | — | — | |
| 37 | Claude-Haiku-3 Anthropic | 73.8% | source | — | — | |
| 38 | Claude 2.1 Anthropic | 73.5% | source | — | — | |
| 39 | Claude Instant Anthropic | 73.4% | source | — | — | |
| 40 | Qwen2.5-7B Alibaba (Qwen) | 72.9% | source | — | — | |
| 41 | Inflection-1 Inflection AI | 72.7% | source | — | — | |
| 42 | Gemma 2 9B Google | 72.1% | source | — | — | |
| 43 | GPT-3.5-turbo OpenAI | 71.4% | source | — | — | |
| 44 | Amazon Nova Micro Amazon | 70.8% | source | — | — | |
| 45 | Falcon-180B Technology Innovation Institute | 70.6% | source | — | — | |
| 46 | Mixtral 8x7B Mistral | 70.6% | source | — | — | |
| 47 | davinci-002 OpenAI | 70.0% | source | — | — | |
| 48 | Gemini 1.0 Pro Google | 70.0% | source | — | — | |
| 49 | Llama 2-70B Meta | 69.9% | source | — | — | |
| 50 | Command R+ Cohere | 69.4% | source | — | — | |
| 51 | PaLM (540B) Google | 69.3% | source | — | — | |
| 52 | Llama 3-8B Meta | 68.8% | source | — | — | |
| 53 | Mistral Large Mistral | 68.8% | source | — | — | |
| 54 | phi-3-mini 3.8B Microsoft | 68.8% | source | — | — | |
| 55 | Mistral Small 4 Mistral | 68.7% | source | — | — | |
| 56 | Qwen1.5-14B Alibaba (Qwen) | 68.6% | source | — | — | |
| 57 | Stable Beluga 2 Stability AI | 68.6% | source | — | — | |
| 58 | Yi-9B Unknown | 68.4% | source | — | — | |
| 59 | Qwen2.5-Coder (1.5B) Alibaba (Qwen) | 68.0% | source | — | — | |
| 60 | Chinchilla Google | 67.5% | source | — | — | |
| 61 | Qwen-14B Alibaba (Qwen) | 66.3% | source | — | — | |
| 62 | Gemma 7B Google | 66.1% | source | — | — | |
| 63 | c4ai-command-r-08-2024 Unknown | 65.2% | source | — | — | |
| 64 | StarCoder 2 15B Hugging Face | 64.1% | source | — | — | |
| 65 | Yi 6B 01.AI | 64.0% | source | — | — | |
| 66 | LLaMA-65B Meta | 63.4% | source | — | — | |
| 67 | Llama 2-34B Meta | 62.6% | source | — | — | |
| 68 | Qwen1.5-7B Alibaba (Qwen) | 62.6% | source | — | — | |
| 69 | Mistral 7B v0.1 Mistral | 62.5% | source | — | — | |
| 70 | Mistral 7B v0.2 Mistral | 62.5% | source | — | — | |
| 71 | DeepSeek-Coder-V2-Lite-Base Unknown | 60.5% | source | — | — | |
| 72 | Gopher (280B) Google | 60.0% | source | — | — | |
| 73 | Mistral 7B v0.3 Mistral | 59.9% | source | — | — | |
| 74 | Baichuan2-13B Baichuan | 59.2% | source | — | — | |
| 75 | LLaMA-33B Meta | 58.7% | source | — | — | |
| 76 | Nemotron-4 15B NVIDIA | 58.7% | source | — | — | |
| 77 | Falcon 2 11B Technology Innovation Institute | 58.4% | source | — | — | |
| 78 | Phi-2 Microsoft | 58.4% | source | — | — | |
| 79 | internlm-chat-20b Unknown | 57.4% | source | — | — | |
| 80 | Falcon-40B Technology Innovation Institute | 56.9% | source | — | — | |
| 81 | Llama 3.2 11B Meta | 56.5% | source | — | — | |
| 82 | Llama 3.1-8B Meta | 56.1% | source | — | — | |
| 83 | Llama 2-13B Meta | 55.6% | source | — | — | |
| 84 | Baichuan2-13B-Chat Unknown | 55.1% | source | — | — | |
| 85 | Baichuan 2-7B Baichuan | 54.2% | source | — | — | |
| 86 | PaLM 62B Unknown | 53.7% | source | — | — | |
| 87 | Baichuan 1-13B Baichuan | 51.6% | source | — | — | |
| 88 | internlm-7b Unknown | 51.0% | source | — | — | |
| 89 | INTELLECT-1 Prime Intellect | 49.9% | source | — | — | |
| 90 | chatglm2-6b Unknown | 47.9% | source | — | — | |
| 91 | MPT-30B MosaicML | 47.9% | source | — | — | |
| 92 | LLaMA-13B Meta | 47.7% | source | — | — | |
| 93 | Llama 2-7B Meta | 45.8% | source | — | — | |
| 94 | Qwen-7B Alibaba (Qwen) | 45.0% | source | — | — | |
| 95 | InstructGPT 175B OpenAI | 43.9% | source | — | — | |
| 96 | Baichuan1-7B Baichuan | 42.3% | source | — | — | |
| 97 | Gemma 2B Google | 42.3% | source | — | — | |
| 98 | Qwen2.5-Coder-0.5B Unknown | 42.0% | source | — | — | |
| 99 | CodeQwen1.5-7B Unknown | 40.5% | source | — | — | |
| 100 | DeepSeek Coder 33B DeepSeek | 39.4% | source | — | — | |
| 101 | StarCoder 2 7B Hugging Face | 38.8% | source | — | — | |
| 102 | Phi-1.5 Microsoft | 37.6% | source | — | — | |
| 103 | StarCoder 2 3B Hugging Face | 36.6% | source | — | — | |
| 104 | DeepSeek Coder 6.7B DeepSeek | 36.4% | source | — | — | |
| 105 | XGen-7B Salesforce | 36.3% | source | — | — | |
| 106 | LLaMA-7B Meta | 35.6% | source | — | — | |
| 107 | Falcon-7B Technology Innovation Institute | 35.0% | source | — | — | |
| 108 | MPT-7B MosaicML | 30.8% | source | — | — | |
| 109 | open_llama_7b Unknown | 29.9% | source | — | — | |
| 110 | Qwen-1_8B Unknown | 28.2% | source | — | — | |
| 111 | RedPajama-INCITE-7B-Base Unknown | 26.3% | source | — | — | |
| 112 | Cerebras-GPT-13B Cerebras Systems | 26.2% | source | — | — | |
| 113 | Dolly 2.0-12b Databricks | 26.2% | source | — | — | |
| 114 | DeepSeek Coder 1.3B DeepSeek | 25.8% | source | — | — | |
| 115 | GPT-J-6B EleutherAI | 25.7% | source | — | — | |
| 116 | opt-13b Unknown | 25.1% | source | — | — |