ErdosBench leaderboard

Research-math reasoning, judged beyond raw solved counts.

ErdosBench evaluates how models behave on research-level, Erdős-inspired mathematical problems: finding decisive obstructions, using known theorems correctly, producing scoped partial progress, and avoiding unsafe solved claims.

The leaderboard below is based on the full 226-problem run with external judge grades and model-specific proof audits. It is not a formal theorem-certification board; it is a correctness-first signal of which systems generate useful, review-worthy mathematical progress.

226Erdős-inspired candidate problems, with provenance and audit metadata
2,034model/problem judge rows across nine scored full-run leaderboards
25four-model A-consensus problems used as high-confidence review anchors
78GPT-5.6 Sol xhigh non-partial resolutions proof-audited

External-judge leaderboard

Grades use a compact A/B/C/D/F/M scale: A means mathematically sound and appropriately scoped; B means useful but incomplete or underpowered; C means plausible with a material gap; D/F mark misleading or contradicted claims; M marks a missing row.

A sound / scoped B useful C material gap D likely wrong F contradicted M missing
Rank Model Coverage Judge grades Strong claims Proof verification read Avg Leaderboard judgment
1
GPT-5.6 Sol xhighBest full-run result so far
226/226rows judged A/B/C/D/F/M: 78 / 142 / 6 / 0 / 0 / 0 78self strong claims 48 verified · 9 literal · 21 conditional · 0 partial-mislabeled · 0 rejected 3.12judge avg, 0–4 Best full-run result so far: largest A-grade yield, full coverage, almost no weak rows, and no rejected strong claims. It materially improves over GPT-5.5 xhigh by tripling A-grade rows and reducing C-grade rows from 40 to 6.
2
Kimi K3 maxStrongest Kimi-family full run
226/226rows judged A/B/C/D/F/M: 40 / 143 / 43 / 0 / 0 / 0 45self strong claims Strong audit: A=40 · B=2 · C=3 · D=0 · F=0; 3 proof-gap/conditional, 2 partial-mislabeled or triage, 0 rejected 2.74judge avg, 0–4 Strongest Kimi-family full run so far. Full coverage, high A-grade yield, no D/F judge rows, and no rejected strong claims. Below GPT-5.6 Sol xhigh, but ahead of GPT-5.5 xhigh and Kimi K2.7 in this audit pass.
3
Muse Spark 1.2Meta frontier model
226/226rows judged A/B/C/D/F/M: 48 / 126 / 47 / 4 / 0 / 1 40self strong claims Strong audit: A=30 · B=6 · C=4 · D=0 · F=0; partials: A=18 · B=120 · C=43 · D=4 · F=0 · M=1; raw fallback=1 2.73judge avg, 0–4 High-yield full run with a clean strong-claim profile and almost no missing output. Ranks just below Kimi K3 and above GPT-5.5 xhigh in this audit pass.
4
GPT-5.5 xhigh / CodexPrevious correctness-adjusted leader
226/226rows judged A/B/C/D/F/M: 27 / 159 / 40 / 0 / 0 / 0 63self strong claims 30 verified · 9 literal · 16 conditional · 8 partial-mislabeled · 0 rejected 2.68judge avg, 0–4 Former leader: full coverage and no D/F judge rows, but Kimi K3 and Muse Spark 1.2 now place ahead through higher A-grade yields and stronger averages.
5
Kimi K2.7 CodeEarlier Kimi-family run
220/226rows judged A/B/C/D/F/M: 32 / 151 / 32 / 3 / 2 / 6 50self strong claims Strong audit: A=30 · B=8 · C=7 · D=3 · F=2; solved-label audit has 7 proof gaps and 4 rejected 2.62judge avg, 0–4 Best at compact constructions and counterexamples, but missing rows and proof-gap/rejected solved labels reduce trust.
6
Claude Opus 4.8 maxBest reviewer and partial-depth model
226/226rows judged A/B/C/D/F/M: 22 / 135 / 69 / 0 / 0 / 0 37self strong claims 9 green natural · 4 green literal · 6 amber partial/overclaimed · 2 red 2.51judge avg, 0–4 Safest broadly: strong proof hygiene and scoped partials, but fewer decisive breakthroughs than GPT or Kimi.
7
GLM-5.2Strong but uneven full-run entrant
222/226rows judged A/B/C/D/F/M: 35 / 95 / 82 / 7 / 3 / 4 33self strong claims Strong audit: A=23 · B=3 · C=3 · D=3 · F=1; partials: 12 A · 92 B · 79 C · 4 D · 2 F 2.402.36 adjusted Strong partial mass and clean core results, but a large gappy middle band and several invalid theorem-scope jumps keep it below Opus.
8
Qwen 3.7 MaxUseful full-coverage corroborator
226/226rows judged A/B/C/D/F/M: 18 / 113 / 91 / 3 / 1 / 0 20self strong claims 12 solved labels: 5 complete · 3 literal/conditional · 2 partial/overclaimed · 2 incorrect/not solved 2.37judge avg, 0–4 Good when it agrees with others, but isolated solved claims are high-risk and proof hygiene is weaker.
9
MiniMax M3Full coverage, weaker proof hygiene
226/226rows judged A/B/C/D/F/M: 15 / 77 / 113 / 12 / 9 / 0 27self strong claims Strong claims: A=9 · B=1 · C=6 · D=5 · F=6; partials: 6 A · 75 B · 107 C · 7 D · 3 F 2.11judge avg, 0–4 Complete coverage and some useful compact results, but C-heavy partials and unreliable solved labels place it below Qwen.

The safest public signal is not raw solved count. ErdosBench separates clean theorem-style proofs, literal or convention-dependent resolutions, conditional arguments, useful partials, and rejected claims.

Why ErdosBench is valuable

Decisive obstruction finding

Does the model spot a hidden divisor bound, coloring invariant, degeneracy, or counterexample that changes the problem?

Known theorem use

Does it deploy the right theorem family without hallucinating scope, constants, or missing hypotheses?

Proof hygiene

Does it distinguish solved, conditional, partial, literal, and rejected claims instead of overmarketing a sketch?

Review yield

Does it produce enough high-quality, review-ready mathematical material to justify expert attention?

Leaderboard read: GPT-5.6 Sol xhigh is the strongest correctness-adjusted full-run model so far.

Kimi K3 is second overall and the strongest Kimi-family full run. Muse Spark 1.2 ranks third with a high-yield, clean strong-claim profile; both place ahead of GPT-5.5 xhigh and Kimi K2.7.

Claude Opus 4.8 remains the best reviewer/partial-depth model.

GLM-5.2 is a strong but uneven entrant after full partial judging, Qwen is best used for corroboration, and MiniMax M3 is a full-coverage lower baseline with weaker solved-label proof hygiene.

Proof audits: why raw solved counts are not enough

Model Scope audited Verification buckets Takeaway
GPT-5.6 Sol xhigh
78 self strong claims 48 verified 9 literal 21 conditional 0 partial-mislabeled 0 rejected Best full-run result so far: 78 A-grade rows, only 6 C-grade rows, and no rejected strong claims.
Kimi K3 max
45 self strong claims 40 A accepted 2 B triage/partial-mislabeled 3 C proof-gap/conditional 0 rejected Strongest Kimi-family audit so far: no D/F judge rows and no rejected strong claims, with five claims correctly held below A pending stronger proof or scope.
Muse Spark 1.2
40 self strong claims; 186 partial-progress rows 30 A strong 6 B strong 4 C strong 0 D/F strong 18 A partial 120 B partial 43 C partial 4 D partial 1 missing/raw fallback Clean strong-claim audit with no rejected solved or counterexample claims. The main risk sits in the middle band: 47 C-grade rows and 4 D-grade partials still require proof closure or scope correction.
GPT-5.5 xhigh / Codex
63 non-partial attempted resolutions30 verified 9 literal 16 conditional 8 partial-mislabeled 0 rejected Former full-run leader; conditional and partial-mislabeled rows still require expert extraction before public solved claims.
Kimi K2.7 Code
30 solved-label claims; 50 strong claims deep-audited6 verified 4 literal 4 conditional 5 partial-mislabeled 7 proof gaps 4 rejected Earlier Kimi-family run; solved labels are high-value but need strict proof-checking.
Claude Opus 4.8 max
21 solved-label claims9 green natural 4 green literal 6 amber 2 red Best reviewer profile: careful partials and caveats, but not self-certifying on solved labels.
GLM-5.2
33 strong claims; 189 partial-progress rows23 A strong 3 B strong 3 C strong 4 D/F strong 12 A partial 92 B partial 79 C partial 6 D/F partial Strong full-run entrant: many useful partials, but theorem-scope mistakes and invalid partials keep proof hygiene below Opus.
Qwen 3.7 Max
12 solved-label claims5 complete 3 literal/conditional 2 partial 2 incorrect/not solved Useful corroborator; isolated solved claims should be treated as high-risk.
MiniMax M3
27 strong claims; 198 partial-progress rows9 A strong 1 B strong 6 C strong 11 D/F strong 6 A partial 75 B partial 107 C partial 10 D/F partial Full coverage, but solved labels are unreliable and the partial profile is C-heavy; best used as lower baseline and failure-mode data.

Model-level diagnosis and post-training opportunities

Model Diagnosis
GPT-5.6 Sol xhigh

Strengths: the best full-run result so far, with decisive obstruction finding, broad theorem deployment, 78 A-grade rows, and no rejected strong claims. Weaknesses: 142 rows remain useful partials, often stopping at one-sided bounds, corrected formulations, reductions, or finite variants; literal and conditional wins are valid but not all equally deep discoveries. Post-training: target the remaining exact asymptotics, uniform analytic number theory, and extremal-stability gaps with stronger theorem-dependency and proof-completion signals.

Kimi K3 max

Strengths: the strongest Kimi-family full run, with decisive obstruction finding, compact counterexamples, strong theorem selection, 40 A-grade rows, and 143 useful B-grade rows. Weaknesses: 43 rows retain material gaps, and Problems 9, 38, and 125 show the main failure mode: a sketched lemma, long case analysis, or conditional construction is promoted to a solved or counterexample label too early. Post-training: reinforce theorem-dependency checks, unconditional-vs-conditional labeling, and a final proof-debt review before allowing strong verdicts.

Muse Spark 1.2

Strengths: a high-yield full-bench entrant with decisive obstruction finding, strong theorem selection, 48 A-grade rows, 126 useful B-grade rows, and 30 accepted strong claims with no D/F strong-claim failures. Weaknesses: the 47 C-grade and 4 D-grade rows cluster around unproved analytic-number-theory uniformity, long case analyses, conditional theorem transfers, and mis-scoped interpretations; one row required a raw-preserving fallback. Post-training: emphasize proof closure, hypothesis and uniformity checks, verdict calibration, and machine-checkable certificate evidence before promoting promising research directions to strong claims.

GPT-5.5 xhigh / Codex

Strengths: previous full-run leader, excellent at decisive obstructions, exact formulas, and literal counterexamples. Weaknesses: many isolated strong claims are conditional, wording-dependent, or partial-mislabeled. Post-training: our audited rows can train stronger verdict calibration: require theorem dependencies, literal-vs-natural statement tags, and a final “what remains unproved?” check before a solved label is allowed.

Kimi K2.7 Code

Strengths: creative earlier Kimi-family run, strong on compact constructions and counterexamples. Weaknesses: solved labels sometimes hide proof gaps, false uniformity assumptions, or theorem-scope drift. Post-training: our accepted-vs-rejected Kimi pairs are ideal for DPO and process-reward training focused on quantifier discipline, uniformity checks, and “creative idea but not yet proof” downgrades.

Claude Opus 4.8 max

Strengths: best reviewer profile, strong gap accounting, and many well-scoped partials. Weaknesses: underclaims short decisive obstructions and sometimes keeps a solved subproblem in “partial” mode. Post-training: our data can teach partial-to-theorem extraction: when a divisor bound, coloring invariant, or literal obstruction already closes the statement, promote it confidently while preserving caveats.

GLM-5.2

Strengths: strong entrant with many real A/B partials and good theorem-aware reductions. Weaknesses: theorem importation is uneven; some partials cite stronger-than-available results or make local-to-global jumps. Post-training: our full partial audit can train theorem-scope verification: state the exact theorem used, check hypotheses, and classify each row as strong partial, useful partial, heuristic, or invalid.

Qwen 3.7 Max

Strengths: useful full-coverage corroborator with many reasonable theorem-family suggestions. Weaknesses: fewer decisive accepted results and a tendency to phrase plausible programs as if they were complete proofs. Post-training: our judge labels can train missing-lemma detection: separate proved statement, conjectural extension, and required external lemma before producing a final answer.

MiniMax M3

Strengths: full coverage and a few good compact obstructions, literal counterexamples, and underclaimed A-level partials. Weaknesses: C-heavy partial profile and unreliable solved labels, with many D/F strong claims caused by wrong theorem scope, wrong recurrence, or wrong problem interpretation. Post-training: our data can train solved-label discipline: demote bad proofs, preserve useful subclaims, and reward explicit statement parsing before invoking a theorem.

Benchmark design

What is measured?

Solving is only one signal. ErdosBench also scores statement debugging, counterexample search, theorem selection, proof-gap detection, finite checks, and scoped partial progress.

Why it matters: these are the skills needed for research assistants, theorem-proving agents, and verifier-backed RLVR loops.

What is protected?

The 226 source records are research candidates, not a public list of certified open problems. Private splits, hidden verifier keys, expert rubrics, and review queues stay sealed.

Public reports show aggregate skill, proof hygiene, and leaderboard movement without burning holdout statements.

1Candidate problem corpus with provenance and risk notes
2Full model runs with structured verdicts and evidence
3External judge grades and proof-verification passes
4Review queues for expert mathematicians and verifier authors
5Training data: SFT, critique, reward-model, and RLVR views

Access

Run ErdosBench on your model.

Ulam can run private evaluations, build domain-specific research-math benchmarks, and convert failures into trainable proof-process data. Contact us for hidden-split access, custom model comparisons, or verifier-backed RLVR environments.