Introducing STEMBench

Today we are releasing STEMBench, a correctness-first evaluation of AI research across biology, chemistry, physics, astronomy, Earth science and engineering. Across ten model runs and 1,200 scored responses, no system met a single full research target. The strongest runs, however, now make supported, meaningful progress on nearly every task they attempt.

Most science benchmarks reward a final answer. Real research rarely has one: progress means narrowing a question, building a defensible argument, and being honest about what is still unresolved. STEMBench is built to measure exactly that. It asks how far a model advances a hard, well-scoped research problem, whether that work holds up to review, whether it can be reproduced, and how clearly the model states what remains open.

STEMBench joins ErdosBench, our evaluation of mathematical research, and is published with a fully open companion set, STEMBench-small. Results, methodology and the public task atlas are on the STEMBench page.

A full STEMBench run is 120 research tasks, 20 in each of six domains. Every task is a scoped research problem with a defined target, not a quiz question with a known answer. A response might propose a discriminating experiment, derive a bound, reconcile conflicting evidence, or show that a question cannot be settled with the data available.

Each response receives an integer progress score from 0 to 4, recording how far it advances the original target:

0. No usable progress. Missing, unusable or unsupported output.
1. Framing or protocol. Useful synthesis or a proposed approach.
2. Meaningful partial progress. Supported work, with consequential obligations still open.
3. Substantial target-level progress. Strong progress on the original task, short of completion.
4. Full-target success. The original target is met.

The scale is deliberately demanding at the top. A careful reduction, derivation or research protocol earns real credit, but only meeting the stated target earns a 4. A fluent answer that quietly changes the question does not.

GPT-6 Astra xhigh, run through Codex with web access, shell execution and a roughly 43.5-hour agent workflow, has the highest complete-run average at 2.809. GPT-6.1 Sol follows at 2.759 under a similar tool-enabled setting. Among closed-book runs, Muse Wonder leads at 2.683, followed by Claude Opus 5.5 at 2.621 and Hy4 Preview at 2.586.

Muse Wonder ranks third overall, behind the two Codex runs, with 116 meaningful partial results and four substantial target-level results. Its strongest work derives non-identifiability conditions, exact counterexamples, and invariant or compactness theorems. All 120 assignments contain usable scientific content, but real measurements, physical validation and global certificates remain consequential gaps.

STEMBench120 scores: models in rows, overall average and six domains in columns. Bold scores are the highest in each column.

Model / setting	Average	Biology	Chemistry	Physics	Astronomy	Earth science	Engineering
GPT-6 Astra xhighCodex · With tools	2.809	2.709	2.777	3.025	2.723	2.726	2.894
GPT-6.1 SolCodex · With tools	2.759	2.656	2.730	2.986	2.692	2.676	2.811
Muse WonderClosed-book	2.683	2.615	2.703	2.714	2.647	2.659	2.759
Claude Opus 5.5Closed-book	2.621	1.865	2.666	2.808	2.759	2.760	2.872
Hy4 PreviewClosed-book	2.586	2.587	2.611	2.521	2.583	2.598	2.615
Mistral Large 4Closed-book	2.501	2.459	2.432	2.431	2.597	2.486	2.602
DeepSeek V4.1 FlashClosed-book	2.490	2.546	2.462	2.202	2.602	2.594	2.538
Gemini 4 ArgonClosed-book	2.444	2.348	2.336	2.083	2.778	2.647	2.471
Qwen3.8-27B NVFP4Closed-book	2.314	2.289	2.517	1.788	2.400	2.516	2.376
MiMo V2.6 ProClosed-book	2.172	2.533	2.498	0.983	2.223	2.623	2.172

Bold marks the highest score in each column. Colors use the same fixed 2.17–2.99 display range as the benchmark page, with lower scores in navy. These are fractional progress scores on a 0–4 scale, not percentages solved.

Three findings matter more than the ranking:

• No full targets yet. None of the 1,200 responses reached level 4. Even level 3 is rare: nine results for GPT-6 Astra xhigh, eight for GPT-6.1 Sol, five for Claude Opus 5.5, four for Muse Wonder, three for Gemini 4 Argon and one for Hy4 Preview.
• Reliability moves the ranking. Gemini 4 Argon has the highest usable-answer mean among closed-book runs: its 105 usable responses average 2.793. Fifteen unusable assignments, five that hit the 65,536-token output cap and ten request failures, pull its complete-run average to 2.444. MiMo V2.6 Pro also lost 20 responses, lowering its complete-run mean to 2.172. Muse Wonder’s 120 usable responses help it lead the closed-book complete-run ranking, even though Opus and Gemini have higher usable-answer means.
• Physics separates the field most. Domain averages there run from 0.983 for MiMo V2.6 Pro to 3.025 for GPT-6 Astra xhigh, the widest spread of any domain.

No single run leads every domain:

The full heatmap, run summaries and integer-level distributions are on the STEMBench page.

The integer level is authoritative. Many responses land at level 2, so we add an experimental quality index, Q, to distinguish stronger and weaker work within levels 1 to 3. It never moves a response out of its band: level 0 stays at 0 and level 4 stays at 4.

Q combines six assessments of the same response:

Scope honesty is scored on purpose. A response that is clear about what it has not shown is more useful to a researcher than one that overstates its result.

Every score passed through an audit. Per-task review-sensitivity allowances of ±0.05, ±0.10 or ±0.20 reflect review confidence; these describe how much a judgment could reasonably move, not statistical confidence intervals. Task-score differences within 0.05 are treated as practical ties.

For STEMBench120 we publish aggregate results only. Task statements, per-task judgments and model submissions stay private, so the benchmark remains useful as a held-out test.

Alongside the private benchmark we are releasing STEMBench-small v1.0.2: 60 research tasks, ten per domain, under the Apache-2.0 licence. It uses the same 0–4 progress scale and ships with a helper for preparing and checking submissions. It is available on Hugging Face and GitHub.

STEMBench-small is a separate release, not a sample of STEMBench120. It ships without baseline scores, solver responses or grades. Anyone can run it under their own setting and report integer levels, full-target outcomes and review coverage separately.

Every task record lays out the same five-stage research path:

1. Parent question. The broad scientific question the task comes from.
2. Known scope. What existing work has already established, and what it has not.
3. Proposed variant. The narrower, testable research target, stated as a proposal rather than a result.
4. Accessible progress. What counts as real partial progress, corresponding to levels 1 and 2.
5. Full target. What a complete answer must establish, corresponding to level 4.

The tasks are deliberately concrete. They ask, for example, whether pulsed meltwater beneath an Antarctic ice shelf can reduce total shelf melting yet increase grounded-ice loss; whether Saturn’s ring waves can reveal a primordial helium gradient; whether steady-state PFAS destruction holds up over startup and shutdown cycles; and whether every central-energy eigenstate of a fixed spin chain looks thermal on growing blocks. Each record names the prior work it builds on, so a model cannot earn credit for rediscovering a published result.

The task atlas on the STEMBench page shows all sixty tasks as animated tiles. Open any tile to follow its research path from parent question to full target, with references.

A few cautions matter as much as the leaderboard itself:

• Run settings differ. GPT-6 Astra xhigh and GPT-6.1 Sol ran through Codex with code execution, web or source retrieval, and multi-day agent workflows; the other eight runs were closed-book API calls with no model-side tools. All models are allowed up to three retries. The comparison is between systems under different resources, not between bare models.
• Decimals are not decisive. The fractional index is experimental. A gap of a few hundredths sits inside review sensitivity and should not be read as a capability difference.
• Reliability counts. The headline average includes every task, with unusable or missing responses scored 0. We also report the mean over usable answers, which isolates answer quality from completion failures.
• Progress is not probability. A score of 2.7 means strong partial progress on average. It is not a percentage solved, nor a probability that a response is correct.

No run has yet met a full STEMBench target. That is the point of the benchmark: the top of the scale is reserved for research that actually closes a problem, and we expect it to stay hard for a while. We will add runs as new systems appear, and publish the audit snapshot and version with each update.

We would like to see STEMBench-small run widely. Download it, evaluate your system under its own setting, and report the integer levels and full-target outcomes alongside your review process.

If you are building models for scientific work, we also run private evaluations, domain comparisons and failure analyses that turn research mistakes into better training data. Get in touch to discuss an evaluation.