SShogo Bench

WeirdML v3

Unusual ML tasks solved by writing and running PyTorch code.

Agent Index weight
0%
shown for context only
Models tested
27
from the benchmark’s leaderboard and reports
Normalised for the index
0 → 0, 1 → 100
Official site

Results

#ModelScoreNormalisedHarnessCost/taskDate
1Claude Fable 5.2 Supported
Anthropic
43.1%claude_code——
2GPT-6 Astra Supported
OpenAI
42.2%codex_cli——
3GPT-6.1 Sol Estimated
OpenAI
38.3%codex_cli——
4Claude Opus 5.5 Supported
Anthropic
31.2%claude_code——
5GPT-6.1 Terra Estimated
OpenAI
28.1%codex_cli——
6Claude Fable 5.1 Supported
Anthropic
26.0%claude_code——
7Gemini 4 Argon Supported
Google
20.3%gemini_cli——
8Claude Sonnet 5.5 Estimated
Anthropic
19.9%claude_code——
9GPT-6 Sol Estimated
OpenAI
19.7%codex_cli——
10Claude Opus 5 Supported
Anthropic
19.0%claude_code——
11Claude Fable 5 Supported
Anthropic
15.6%claude_code——
12Muse Wonder Estimated
Meta
15.1%codex_cli——
13GPT-5.6 Sol Supported
OpenAI
15.0%codex_cli——
14Claude Haiku 5.5 Estimated
Anthropic
11.7%claude_code——
15Kimi K3.1 Pro
Moonshot AI
9.9%kimi_code——
16DeepSeek V4.1 Flash 1017 Estimated
DeepSeek
9.1%codex_cli——
17Grok 4.7 Estimated
xAI
8.5%codex_cli——
18GPT-6 Luna
OpenAI
7.9%codex_cli——
19Kimi K3 Supported
Moonshot AI
7.5%kimi_code——
20Gemini 3.8 Flash Supported
Google
7.4%gemini_cli——
21Muse Spark 1.3 Estimated
Meta
6.7%codex_cli——
22Claude Opus 4.5 (latest) Supported
Anthropic
6.5%claude_code——
23DeepSeek V4.1 Flash Estimated
DeepSeek
5.9%opencode——
24GPT-5.6 Luna Estimated
OpenAI
5.2%codex_cli——
25GPT-5.2 Supported
OpenAI
5.0%codex_cli——
26Mistral Large 4
Mistral
4.2%opencode——
27Inkling Estimated
thinkingmachines
3.9%opencode——