Synthetic olympiad problems for measuring reasoning, not recall.
SimoBench, the Synthetic International Math Olympiad Bench, is a compact 126-problem benchmark of synthetic olympiad-style mathematics problems for evaluating mathematical reasoning across frontier, small, and mid-sized language models.
The benchmark keeps the spirit and difficulty profile of IMO-style reasoning while moving away from directly memorized contest statements. Each problem is a synthetic variant inspired by the mechanism of an IMO problem, selected to be standalone, interesting, and challenging.
Judge-graded leaderboard
Every submitted proof is scored from 0 to 7 against the reference solution. The ranking is the total over all 126 problems, rather than a model's self-reported solved label.
| Rank | Model | Coverage | Total score | Mean | Full solves | Strong proofs | Substantial progress | Zero scores | Leaderboard read |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.5 xhighBest overall proof conversion |
126/126rows judged | 752of 882 · 85.26% | 5.97out of 7 | 59score-7 proofs | 103scores 5–7 | 118scores 4–7 | 0no-credit rows | Best overall conversion of correct ideas into complete, concise proofs. |
| 2 | Ling-2.6Highest strong-proof count |
126/126rows judged | 746of 882 · 84.58% | 5.92out of 7 | 52score-7 proofs | 111scores 5–7 | 114scores 4–7 | 0no-credit rows | Very high coverage and many complete proofs, with several costly wrong classifications. |
| 3 | HY3-previewNear-leader proof depth |
126/126rows judged | 743of 882 · 84.24% | 5.90out of 7 | 53score-7 proofs | 104scores 5–7 | 114scores 4–7 | 0no-credit rows | Strong proof depth and near-leader total; partials are concentrated on the hardest rows. |
| 4 | Command-A-PlusGood solves, costly blank outputs |
126/126rows judged | 559of 882 · 63.38% | 4.44out of 7 | 38score-7 proofs | 79scores 5–7 | 83scores 4–7 | 25no-credit rows | Good substantive solves, but 25 answer-only or status-only submissions receive zero proof credit. |
| 5 | DeepSeek-V4-FlashHigh ceiling, many empty outputs |
126/126rows judged | 558of 882 · 63.27% | 4.43out of 7 | 42score-7 proofs | 84scores 5–7 | 87scores 4–7 | 38no-credit rows | High ceiling on completed generations; 38 empty failed outputs sharply reduce the total. |
| 6 | MiniMax-M3Strong completions, many empty outputs |
126/126rows judged | 500of 882 · 56.69% | 3.97out of 7 | 39score-7 proofs | 73scores 5–7 | 75scores 4–7 | 44no-credit rows | Many strong completed proofs, offset by 44 empty failed outputs. |
| 7 | Ulam-1-smallBest original small model |
126/126rows judged | 411of 882 · 46.60% | 3.26out of 7 | 25score-7 proofs | 33scores 5–7 | 48scores 4–7 | 2no-credit rows | Best of the original small-model pair; more substantial 4–6 point proofs. |
| 8 | VibeThinker-3bCompact 3B reasoning baseline |
126/126rows judged | 400of 882 · 45.35% | 3.17out of 7 | 25score-7 proofs | 30scores 5–7 | 41scores 4–7 | 1no-credit rows | Same full-solve count as Ulam, but fewer substantial partial proofs. |
All 1,008 rows are scored. Valid counterexamples receive full credit when they resolve a statement as written; empty, answer-only, and status-only responses receive no proof credit. The score retains correct work in truncated proofs instead of collapsing them to a binary result.
Leaderboard read: GPT-5.5 xhigh leads the completed board with the strongest conversion of correct ideas into concise, complete proofs.
Ling-2.6 and HY3-preview are close behind on total score and proof depth. For the lower-ranked runs, empty or answer-only outputs are a major differentiator, not just the quality of completed proofs.
Scoring rubric
SimoBench follows an olympiad-style 0–7 grading scale. For each model, the benchmark score is the sum over all 126 problems.
Primary metrics
Total score: sum of 126 problem grades, maximum 882.
Mean score: total divided by 126, reported on the 0–7 scale.
Full solves: number of problems graded 7.
Secondary metrics
Substantial progress: number of problems graded at least 4.
Pass@k: optional when sampling multiple attempts per problem.
Cost and speed: average tokens and wall-clock time per problem.
Why SimoBench is valuable
Reasoning over recall
Original IMO problems are widely available in training data, solution archives, forum discussions, tutorials, and benchmark reports. SimoBench moves evaluation away from direct memorization.
Olympiad-style structure
The problems preserve mechanisms from high-quality contest mathematics while changing the exact statements and surface form.
Compact and repeatable
With 126 problems, SimoBench is small enough to run frequently while still hard enough to reveal reasoning gaps.
Solution-backed
Every selected benchmark problem has a reference solution, enabling human, assisted, or judge-model grading.
Release framing: SimoBench is a 126-problem synthetic olympiad benchmark for testing mathematical reasoning in small and mid-sized language models.
It is built from 1,260 generated variants inspired by IMO problem mechanisms, with one manually selected problem for each IMO problem slot from 2005 to 2025.
The public benchmark file hides origin metadata and includes only problem IDs and statements; matching reference solutions are provided separately.
Benchmark design
What is tested?
SimoBench covers algebra, number theory, geometry, combinatorics, inequalities, functional equations, games, graphs, and discrete processes.
The tasks ask models to parse a new statement, identify the hidden structure, and build a proof or computation.
What is hidden?
The benchmark-facing file removes the source IMO year, original problem number, variant number, source title, and inspiration metadata.
This prevents prompts from handing the model a strong retrieval cue such as “inspired by IMO 2017 Problem 4.”
| Design choice | Implementation | Evaluation purpose |
|---|---|---|
| One problem per slot | 21 years × 6 problems = 126 benchmark items. | Broad IMO-style coverage without making the benchmark too large to run often. |
| Manual selection | One final problem selected from 10 variants for each IMO slot. | Favor coherent, standalone, interesting, and challenging statements with usable reference solutions. |
| Public file stripped | Only problem_id and problem_statement are shown to solvers. |
Reduce origin shortcuts and contamination-style retrieval. |
| Stable random order | Fixed seed: SimoBench-v1. |
Make experiments reproducible while keeping adjacent problems well mixed. |
Recommended evaluation protocol
Use SimoBench.json as the solver input. Keep reference solutions and all origin metadata out of the model prompt.
Solve the following olympiad-style problem. Provide a rigorous proof.
Problem:
{problem_statement}Input file
Use only the benchmark-facing problem file with problem IDs and statements.
No metadata leaks
Do not include the source IMO year, original problem number, variant number, source title, or internal SIMO metadata.
Grade reasoning
For proof problems, grade the reasoning rather than only the conclusion. For answer-only problems, require both final answer and justification.
Release files
SimoBench.json
Public benchmark input: 126 items with problem_id and problem_statement.
SimoBench-problems.json
Identical public alias for the benchmark input file.
SimoBench-solutions.json
Answer-key file in the same order, adding the reference solution field.
| File group | Files | Use during evaluation |
|---|---|---|
| Benchmark inputs | SimoBench.json, SimoBench-problems.json |
Safe to provide to the model as the problem source. |
| Benchmark solutions | SimoBench-solutions.json |
Use for human or assisted grading, but never include in the solver prompt. |
| Audit and provenance | SIMO-problems.json, SIMO-solutions.json, problems.json, solutions.json |
Useful for dataset maintenance; should not be shown to models during benchmark evaluation. |
Caveats and reproducibility
Caveats
SimoBench is synthetic. That is a strength for evaluation, but the benchmark should be treated as a research artifact rather than an official competition archive.
The current benchmark has reference solutions, but not machine-checkable formal proofs. Grading still requires mathematical judgment.
Reproducibility
The benchmark files can be regenerated from the selected SIMO files with node scripts/create-simobench.mjs.
The selected SIMO files are generated from the TeX source pool with node scripts/prepare-json.mjs.
Access
Run SimoBench on your model.
Ulam can run private evaluations, compare model families under the same 0–7 scoring rubric, and convert failures into trainable proof-process data for mathematical reasoning systems.