SShogo Bench

MMLU

Multiple-choice questions across 57 subjects. Saturated at the frontier.

Agent Index weight
0%
shown for context only
Models tested
116
from the benchmark’s leaderboard and reports
Normalised for the index
0.25 → 0, 1 → 100
Official site

Results

#ModelScoreNormalisedHarnessCost/taskDate
1GPT-4o (2024-08-06) Supported
OpenAI
88.1%source——
2Claude-Sonnet-3.5 Estimated
Anthropic
87.3%source——
3DeepSeek-V3
DeepSeek
87.2%source——
4Gemini 1.5 Pro (Feb 2024)
Google
86.9%source——
5GPT-4
OpenAI
86.4%source——
6Llama 3.3 70B
Meta
86.3%source——
7Qwen2.5-72B Estimated
Alibaba (Qwen)
85.3%source——
8Phi 4
Microsoft
84.8%source——
9Claude 3 Opus Estimated
Anthropic
84.6%source——
10Llama 3.1-405B
Meta
84.5%source——
11Qwen2-72B
Alibaba (Qwen)
82.4%source——
12Amazon Nova Pro
Amazon
82.0%source——
13GPT-4o mini
OpenAI
81.8%source——
14GPT-4 Turbo
OpenAI
81.3%source——
15Llama 3.2 90B
Meta
80.3%source——
16Llama 3.1-70B
Meta
80.1%source——
17Mistral Large 2 (Nov 2024)
Mistral
80.0%source——
18Qwen2.5-14B
Alibaba (Qwen)
79.9%source——
19Gemini-2.0-Flash
Google
79.7%source——
20Llama 3-70B
Meta
79.3%source——
21Yi-Large
01.AI
79.3%source——
22Qwen2.5-Coder-32B
Alibaba (Qwen)
79.1%source——
23Claude 2
Anthropic
78.5%source——
24DeepSeek-V2 (MoE-236B, May 2024)
DeepSeek
78.4%source——
25phi-3-medium 14B
Microsoft
78.0%source——
26Gemini 1.5 Flash (Sep 2024)
Google
77.9%source——
27Mixtral 8x22B
Mistral
77.8%source——
28Amazon Nova Lite
Amazon
77.0%source——
29Claude 1.3
Anthropic
77.0%source——
30Yi-34B
01.AI
76.3%source——
31Claude 3 Sonnet
Anthropic
75.9%source——
32Gemma 2 27B
Google
75.7%source——
33phi-3-small 7.4B
Microsoft
75.7%source——
34Qwen2.5-Coder-14B
Unknown
75.2%source——
35Qwen1.5-32B
Alibaba (Qwen)
74.4%source——
36Claude 3.5 Haiku
Anthropic
74.3%source——
37Claude-Haiku-3
Anthropic
73.8%source——
38Claude 2.1
Anthropic
73.5%source——
39Claude Instant
Anthropic
73.4%source——
40Qwen2.5-7B
Alibaba (Qwen)
72.9%source——
41Inflection-1
Inflection AI
72.7%source——
42Gemma 2 9B
Google
72.1%source——
43GPT-3.5-turbo
OpenAI
71.4%source——
44Amazon Nova Micro
Amazon
70.8%source——
45Falcon-180B
Technology Innovation Institute
70.6%source——
46Mixtral 8x7B
Mistral
70.6%source——
47davinci-002
OpenAI
70.0%source——
48Gemini 1.0 Pro
Google
70.0%source——
49Llama 2-70B
Meta
69.9%source——
50Command R+
Cohere
69.4%source——
51PaLM (540B)
Google
69.3%source——
52Llama 3-8B
Meta
68.8%source——
53Mistral Large
Mistral
68.8%source——
54phi-3-mini 3.8B
Microsoft
68.8%source——
55Mistral Small 4
Mistral
68.7%source——
56Qwen1.5-14B
Alibaba (Qwen)
68.6%source——
57Stable Beluga 2
Stability AI
68.6%source——
58Yi-9B
Unknown
68.4%source——
59Qwen2.5-Coder (1.5B)
Alibaba (Qwen)
68.0%source——
60Chinchilla
Google
67.5%source——
61Qwen-14B
Alibaba (Qwen)
66.3%source——
62Gemma 7B
Google
66.1%source——
63c4ai-command-r-08-2024
Unknown
65.2%source——
64StarCoder 2 15B
Hugging Face
64.1%source——
65Yi 6B
01.AI
64.0%source——
66LLaMA-65B
Meta
63.4%source——
67Llama 2-34B
Meta
62.6%source——
68Qwen1.5-7B
Alibaba (Qwen)
62.6%source——
69Mistral 7B v0.1
Mistral
62.5%source——
70Mistral 7B v0.2
Mistral
62.5%source——
71DeepSeek-Coder-V2-Lite-Base
Unknown
60.5%source——
72Gopher (280B)
Google
60.0%source——
73Mistral 7B v0.3
Mistral
59.9%source——
74Baichuan2-13B
Baichuan
59.2%source——
75LLaMA-33B
Meta
58.7%source——
76Nemotron-4 15B
NVIDIA
58.7%source——
77Falcon 2 11B
Technology Innovation Institute
58.4%source——
78Phi-2
Microsoft
58.4%source——
79internlm-chat-20b
Unknown
57.4%source——
80Falcon-40B
Technology Innovation Institute
56.9%source——
81Llama 3.2 11B
Meta
56.5%source——
82Llama 3.1-8B
Meta
56.1%source——
83Llama 2-13B
Meta
55.6%source——
84Baichuan2-13B-Chat
Unknown
55.1%source——
85Baichuan 2-7B
Baichuan
54.2%source——
86PaLM 62B
Unknown
53.7%source——
87Baichuan 1-13B
Baichuan
51.6%source——
88internlm-7b
Unknown
51.0%source——
89INTELLECT-1
Prime Intellect
49.9%source——
90chatglm2-6b
Unknown
47.9%source——
91MPT-30B
MosaicML
47.9%source——
92LLaMA-13B
Meta
47.7%source——
93Llama 2-7B
Meta
45.8%source——
94Qwen-7B
Alibaba (Qwen)
45.0%source——
95InstructGPT 175B
OpenAI
43.9%source——
96Baichuan1-7B
Baichuan
42.3%source——
97Gemma 2B
Google
42.3%source——
98Qwen2.5-Coder-0.5B
Unknown
42.0%source——
99CodeQwen1.5-7B
Unknown
40.5%source——
100DeepSeek Coder 33B
DeepSeek
39.4%source——
101StarCoder 2 7B
Hugging Face
38.8%source——
102Phi-1.5
Microsoft
37.6%source——
103StarCoder 2 3B
Hugging Face
36.6%source——
104DeepSeek Coder 6.7B
DeepSeek
36.4%source——
105XGen-7B
Salesforce
36.3%source——
106LLaMA-7B
Meta
35.6%source——
107Falcon-7B
Technology Innovation Institute
35.0%source——
108MPT-7B
MosaicML
30.8%source——
109open_llama_7b
Unknown
29.9%source——
110Qwen-1_8B
Unknown
28.2%source——
111RedPajama-INCITE-7B-Base
Unknown
26.3%source——
112Cerebras-GPT-13B
Cerebras Systems
26.2%source——
113Dolly 2.0-12b
Databricks
26.2%source——
114DeepSeek Coder 1.3B
DeepSeek
25.8%source——
115GPT-J-6B
EleutherAI
25.7%source——
116opt-13b
Unknown
25.1%source——