Ulam / Scientific reasoning evaluation

STEMBench.

Research progress.
Across six STEM domains.

A correctness-first evaluation of AI research across biology, chemistry, physics, astronomy, Earth science, and engineering. Compare supported scientific progress, reproducibility, and how honestly a model states what remains unresolved.

Explore domain scores ↓ How scoring works Public test set ↓ ErdosBench ↗
120tasks per full run
6domains · 20 tasks each
10runs · 1,200 scored responses
0full-target successes across all runs
01 / Domain comparison

A field-by-field view.

Experimental fractional progress scores, from 0 to 4. Higher is better.

STEMBench120 · v1.0.0Audit snapshot ·

10 of 10 runs · 20 tasks per domain

2.172.99+
Below scale (< 2.17) Bold = highest in column among visible runs

Each domain contains 20 tasks. Values follow the published audit aggregates. Scores are on a 0–4 scale; color focuses on the 2.17–2.99 range to show differences clearly. Lower scores use navy; scores above the display range use yellow. This color range stays fixed when filtering. The average includes every task, including zero-score attempts. Fractional scores describe progress, not a percentage solved or a probability of correctness.

02 / Full-run results

The score, with its context.

Ranked by overall fractional average within the selected run setting.

0 · No usable progress 1 · Framing / protocol 2 · Meaningful partial 3 · Substantial progress 4 · Full target

Integer progress scores remain authoritative. Usable-answer means exclude assignments without usable content; complete-run means retain their zero scores. All ten runs have zero full-target successes. Small differences in the experimental fractional average should be read alongside the progress distribution, run setting, and review uncertainty.

03 / Scoring & audit

Credit for work that holds up.

Progress, scientific support, and remaining obligations are assessed separately.

Integer progress comes first

The 0–4 judgment records how far the response advances the original target. A useful reduction, derivation, or research protocol can earn credit while the full task remains unresolved.

  1. 0No usable progress. Missing, unusable, or unsupported output.
  2. 1Framing or protocol. Useful synthesis or a proposed approach.
  3. 2Meaningful partial progress. Supported work with consequential obligations still open.
  4. 3Substantial target-level progress. Strong progress on the original task, short of full completion.
  5. 4Full-target success. The original target is met.

Fractional scores add resolution

An experimental quality index, Q, distinguishes work within levels 1–3. It preserves each task’s integer band and its full-target outcome. Level 0 stays at 0; level 4 stays at 4.

Fractional score = s + 0.95 × Q
for integer score s ∈ {1, 2, 3}
Supported achievement strength
30%
Original-obligation coverage proxy
20%
Verification and reproducibility
15%
Scientific support
15%
Validity surviving decisive gaps
10%
Scope honesty and content integrity
10%

Reading precision responsibly. The audit uses per-task review-sensitivity allowances of ±0.05, ±0.10, or ±0.20 according to review confidence. These are review-sensitivity ranges, not statistical confidence intervals for model means. The comparison report treats task-score differences within 0.05 as practical ties. The fractional index is experimental; decimal precision does not establish a decisive capability gap.

For STEMBench120, this page publishes aggregate results only. Its task statements, per-task judgments, and model submissions remain outside the public comparison. The public STEMBench-small set below is a separate release.

04 / Public test set

STEMBench-small: sixty tasks, in the open.

A public test case with ten research tasks per domain, the same 0–4 progress scale, and a helper for preparing and checking submissions. Open any tile to follow its research path from the parent question to the full target.

Hugging Face dataset ↗ GitHub repository ↗
STEMBench-small · v1.0.260 tasks · Apache-2.0
1Parent question 2Known scope 3Proposed variant 4Accessible progress 5Full target

Each animation is a schematic of the system its task studies, not data or a result. The track under a tile marks the five stages that every task record sets out; level 4 on the scale above corresponds to the full target.

Evaluate your model across STEM.

Work with Ulam on private evaluations, domain comparisons, and evidence that helps turn research failures into better training data.

Discuss an evaluation ↗