SShogo Bench

Aider Polyglot

225 hard Exercism exercises in six languages, edited through Aider.

Agent Index weight
0%
shown for context only
Models tested
45
from the benchmark’s leaderboard and reports
Normalised for the index
0 → 0, 1 → 100
Official site

Results

#ModelScoreNormalisedHarnessCost/taskDate
1GPT-5 Supported
OpenAI
88.0%source——
2o3-pro
OpenAI
84.9%source——
3Gemini 2.5 Pro Supported
Google
83.1%source——
4o3 Supported
OpenAI
81.3%source——
5Grok 4 Supported
xAI
79.6%source——
6DeepSeek V3.2 Supported
DeepSeek
74.2%source——
7Claude-Opus-4 Supported
Anthropic
72.0%source——
8o4-mini Estimated
OpenAI
72.0%source——
9DeepSeek-R1 Estimated
DeepSeek
71.4%source——
10Claude 3.7 Sonnet Supported
Anthropic
64.9%source——
11o1 Estimated
OpenAI
61.7%source——
12Claude Sonnet 4 Supported
Anthropic
61.3%source——
13o3-mini Estimated
OpenAI
60.4%source——
14Qwen3 235B A22B
Alibaba (Qwen)
59.6%source——
15Qwen3 235B A22B Instruct 2507
Alibaba (Qwen)
59.6%source——
16Kimi K2 0905 Estimated
Moonshot AI
59.1%source——
17DeepSeek V3 0324
DeepSeek
55.1%source——
18Gemini 2.5 Flash Estimated
Google
55.1%source——
19Grok 3
xAI
53.3%source——
20GPT-4.1
OpenAI
52.4%source——
21Claude-Sonnet-3.5 Estimated
Anthropic
51.6%source——
22Grok-3 mini
xAI
49.3%source——
23DeepSeek-V3
DeepSeek
48.4%source——
24GPT-4o (2024-08-06) Supported
OpenAI
45.3%source——
25GPT-4.5
OpenAI
44.9%source——
26GPT OSS 120B Supported
OpenAI
41.8%source——
27Qwen3 32B
Alibaba (Qwen)
40.0%source——
28Gemini-2.0-Flash
Google
38.2%source——
29Gemini 2.0 Pro
Google
35.6%source——
30o1-mini
OpenAI
32.9%source——
31GPT-4.1 mini
OpenAI
32.4%source——
32Claude 3.5 Haiku
Anthropic
28.0%source——
33Qwen2.5-Max
Alibaba (Qwen)
21.8%source——
34Qwq 32B
Alibaba (Qwen)
20.9%source——
35Gemini 2.0 Flash Thinking (Jan 2025)
Google
18.2%source——
36DeepSeek-V2.5 (Sep 2024)
DeepSeek
17.8%source——
37Qwen2.5 Coder 32B Instruct
Alibaba (Qwen)
16.4%source——
38Llama 4 Maverick
Meta
15.6%source——
39Yi-Lightning
01.AI
12.9%source——
40Cohere Command A
Cohere
12.0%source——
41Codestral 25.08
Mistral
11.1%source——
42openhands-lm-32b-v0.1
Unknown
10.2%source——
43GPT-4.1 nano
OpenAI
8.9%source——
44Gemma 3 27B
Google
4.9%source——
45GPT-4o mini
OpenAI
3.6%source——