Synthetic olympiad problems for measuring reasoning, not recall.
SimoBench, the Synthetic International Math Olympiad Bench, is a compact 126-problem benchmark of synthetic olympiad-style mathematics problems for evaluating mathematical reasoning in small and mid-sized language models.
The benchmark keeps the spirit and difficulty profile of IMO-style reasoning while moving away from directly memorized contest statements. Each problem is a synthetic variant inspired by the mechanism of an IMO problem, selected to be standalone, interesting, and challenging.
Leaderboard coming soon
No public leaderboard yet.
SimoBench will use a numerical leaderboard format. Each model run will receive a score from 0 to 7 on each of the 126 problems, then a proper aggregate score over the whole benchmark.
The headline score will report the total out of 882 points, the mean score out of 7, the number of full solves, and the number of problems where the model makes substantial progress.
Leaderboard in preparation| Rank | Model | Coverage | Total score | Mean | Full solves | Substantial progress | Notes |
|---|---|---|---|---|---|---|---|
| — | Coming soon Public model results have not been posted yet. |
126 / 126 planned | 0–882 | 0–7 | score 7 count | score ≥ 4 count | Rows will be added after standardized runs and grading against the reference solutions. |
The goal is to reward mathematical progress without collapsing everything into a binary solved/not-solved label. A model can receive partial credit for correct key observations, a nearly complete proof, or a complete solution with minor gaps.
Scoring rubric
SimoBench follows an olympiad-style 0–7 grading scale. For each model, the benchmark score is the sum over all 126 problems.
Primary metrics
Total score: sum of 126 problem grades, maximum 882.
Mean score: total divided by 126, reported on the 0–7 scale.
Full solves: number of problems graded 7.
Secondary metrics
Substantial progress: number of problems graded at least 4.
Pass@k: optional when sampling multiple attempts per problem.
Cost and speed: average tokens and wall-clock time per problem.
Why SimoBench is valuable
Reasoning over recall
Original IMO problems are widely available in training data, solution archives, forum discussions, tutorials, and benchmark reports. SimoBench moves evaluation away from direct memorization.
Olympiad-style structure
The problems preserve mechanisms from high-quality contest mathematics while changing the exact statements and surface form.
Compact and repeatable
With 126 problems, SimoBench is small enough to run frequently while still hard enough to reveal reasoning gaps.
Solution-backed
Every selected benchmark problem has a reference solution, enabling human, assisted, or judge-model grading.
Release framing: SimoBench is a 126-problem synthetic olympiad benchmark for testing mathematical reasoning in small and mid-sized language models.
It is built from 1,260 generated variants inspired by IMO problem mechanisms, with one manually selected problem for each IMO problem slot from 2005 to 2025.
The public benchmark file hides origin metadata and includes only problem IDs and statements; matching reference solutions are provided separately.
Benchmark design
What is tested?
SimoBench covers algebra, number theory, geometry, combinatorics, inequalities, functional equations, games, graphs, and discrete processes.
The tasks ask models to parse a new statement, identify the hidden structure, and build a proof or computation.
What is hidden?
The benchmark-facing file removes the source IMO year, original problem number, variant number, source title, and inspiration metadata.
This prevents prompts from handing the model a strong retrieval cue such as “inspired by IMO 2017 Problem 4.”
| Design choice | Implementation | Evaluation purpose |
|---|---|---|
| One problem per slot | 21 years × 6 problems = 126 benchmark items. | Broad IMO-style coverage without making the benchmark too large to run often. |
| Manual selection | One final problem selected from 10 variants for each IMO slot. | Favor coherent, standalone, interesting, and challenging statements with usable reference solutions. |
| Public file stripped | Only problem_id and problem_statement are shown to solvers. |
Reduce origin shortcuts and contamination-style retrieval. |
| Stable random order | Fixed seed: SimoBench-v1. |
Make experiments reproducible while keeping adjacent problems well mixed. |
Recommended evaluation protocol
Use SimoBench.json as the solver input. Keep reference solutions and all origin metadata out of the model prompt.
Solve the following olympiad-style problem. Provide a rigorous proof.
Problem:
{problem_statement}Input file
Use only the benchmark-facing problem file with problem IDs and statements.
No metadata leaks
Do not include the source IMO year, original problem number, variant number, source title, or internal SIMO metadata.
Grade reasoning
For proof problems, grade the reasoning rather than only the conclusion. For answer-only problems, require both final answer and justification.
Release files
SimoBench.json
Public benchmark input: 126 items with problem_id and problem_statement.
SimoBench-problems.json
Identical public alias for the benchmark input file.
SimoBench-solutions.json
Answer-key file in the same order, adding the reference solution field.
| File group | Files | Use during evaluation |
|---|---|---|
| Benchmark inputs | SimoBench.json, SimoBench-problems.json |
Safe to provide to the model as the problem source. |
| Benchmark solutions | SimoBench-solutions.json |
Use for human or assisted grading, but never include in the solver prompt. |
| Audit and provenance | SIMO-problems.json, SIMO-solutions.json, problems.json, solutions.json |
Useful for dataset maintenance; should not be shown to models during benchmark evaluation. |
Caveats and reproducibility
Caveats
SimoBench is synthetic. That is a strength for evaluation, but the benchmark should be treated as a research artifact rather than an official competition archive.
The current benchmark has reference solutions, but not machine-checkable formal proofs. Grading still requires mathematical judgment.
Reproducibility
The benchmark files can be regenerated from the selected SIMO files with node scripts/create-simobench.mjs.
The selected SIMO files are generated from the TeX source pool with node scripts/prepare-json.mjs.
Access
Run SimoBench on your model.
Ulam can run private evaluations, compare model families under the same 0–7 scoring rubric, and convert failures into trainable proof-process data for mathematical reasoning systems.