Datasets · Benchmarks · Verifiers

Find where reasoning fails.
Train the fix.

Ulam builds research-level problem sets, benchmarks and reasoning datasets that preserve the evidence between prompt and answer: attempts, proof units, tool calls, negative traces, critiques, repairs, preferences, and verifier outcomes. Inspect our public releases or commission reviewed corpora, private holdouts, reward-ready exports, and original mathematics problem sets from AIME through open-research difficulty.

Explore Math RL See ErdosBench Discuss data access

100,000+internally created, unpublished private math problems

< 30%below 30% pass@1 from SOTA models on our STEM RL tasks

1,000+research-level trajectories

20,000+tiered OlympiadNet records across reviewed and candidate layers

Flagship reasoning datasets and private mathematics problems

From inspectable public schemas to large candidate layers and reviewed private packs—built for training, criticism, verification, and evaluation rather than answer-only supervision. Our original problem catalogue spans exact-answer competition mathematics, graduate and PhD problems, verifier-backed RL tasks, and open research.

AIME · Olympiad · Graduate · Research

Private Mathematics Problems

100,000+ internally created problems

Original, unpublished private mathematics problems for training and evaluation—from AIME and olympiad level through graduate, PhD, and research-level mathematics. Deliveries can mix exact-answer problems, proof tasks, math problems with executable verifiers for RL, and entirely open questions.

AIME++ samples exact-answer tasks across AIME, AIME Hard, graduate, and researcher tiers.
Math RL Tasks samples graduate- and research-level math problems packaged with executable verifiers and RL environments.
SOTA Math samples open-research problems, milestone-based tasks, and bounded counterexample searches.

Private catalogue · Public samples

Discuss problem-set access →

Scientific agents · RL · Private evaluation

Terminal-Bench Science-derived

1,000 scientific-agent environments

Hard scientific tasks spanning numerical software, experimental design, inference, uncertainty estimation, system identification, control, optimization, and robustness. Built for scientific-agent RL and evaluation across unfamiliar inputs and conditions.

Hidden numerical inputs and scientific-campaign evaluators score performance and integrity. Every task includes a solution and a short calibrated evaluation with GPT-5.6 Sol or GPT-6 Astra.

Scientific tasks · Verifier-backed rewards

Explore science RL sample →

Research proof processes

Verified Research Reasoning

1,000+ trajectory program

Proof-process data that records where research reasoning becomes valid, conditional, incomplete, or wrong: dependency-linked proof units, gaps, first-bad-step labels, negative traces, rewards, critiques, repairs, preferences, and adversarial tests.

The public inspection release includes 3 canonical trajectories, 29 PVUs, 13 negative traces, 17 evaluation tasks, schema, transcripts, and validators. Machine-checkability applies only where indicated.

Public inspection release · Licensed packs

Inspect Verified Reasoning →

Olympiad proofs · SFT · RLVR

OlympiadNet-Math

20,000+ tiered records

Proof and final-answer data with source solutions kept separate from model attempts, plus proof units, negative traces, preferences, adversarial tests, review queues, and promotion gates.

Strict RLVR rows are an explicitly promoted subset—not a label for the whole corpus. The public release provides 10 inspection examples, schema, and quality-gate definitions.

Public inspection release · Tiered private corpus

Inspect OlympiadNet →

Benchmarks and environments

The evaluation layer shows whether a model can use the data: reason under uncertainty, construct valid proofs, and act correctly through tools.

Research benchmark

ErdősBench

226 research candidates testing obstruction finding, counterexamples, finite experiments, theorem use, proof gaps, partial progress, and calibrated claims.

14-problem public smoke test; private full evaluation.

Explore ErdősBench →

STEM research benchmark

STEMBench

120 research-level tasks across biology, chemistry, physics, astronomy, Earth science, and engineering. Compare models through domain heatmaps and audited scientific progress, reproducibility, and scope honesty.

Six domains, 20 tasks each, with progress scores from 0 to 4.

Explore STEMBench →

Open Math RL Environments

Math RL Tasks

Graduate- and research-level mathematics problems packaged as executable RL environments, with exact, structured-output, witness, construction, and certificate verifiers.

Open samples spanning foundational, medium-to-hard, and frontier research-math task families.

Explore Math RL Tasks →

Training signals beyond final answers

Use inspectable public data, prover-backed traces, or a private build targeted to your model's actual failures.

Benchmarks RL

Benchmark-inspired RL tasks

Clean-room task families turn Terminal-Bench, Terminal-Bench Science, GPQA Diamond, Humanity's Last Exam, CritPt, SciCode, and GDPVal capability surfaces into executable environments.

These are synthetic or task-derived environments. Rewards use exact checks, schemas, certificates, simulators, or stateful verifiers by family.

Browse task families →

Formal proof traces

UlamAI Prover + LLM

LLMs propose tactics or proof edits; Lean checks them. Retain proof states, premise retrieval, accepted and rejected actions, errors, repairs, backtracking, and terminal outcomes.

Useful for tactic SFT, critics, proof repair, preference data, RLVR, and regression. Formal status applies to Lean-accepted proof steps.

View UlamAI Prover →

Private and targeted

Custom failure-driven builds

Bring representative attempts, tool logs, proofs, or evaluation results. Ulam localizes recurring failures and returns positives, negatives, critiques, repairs, preference pairs, verifier contracts, and sealed holdouts.

Delivered as JSONL or Parquet with stable IDs, provenance, splits, quality status, and training-view exports.

One improvement loop

Benchmarks reveal the failure. Verifiers make it measurable. Training data turns the repair into a repeatable signal.

problem → attempt/action → observation → verifier verdict → first failure → critique/repair → training export

01 MeasureRun ErdősBench, SimoBench, STEMBench, or customer tasks.

02 DiagnoseLocate the first meaningful error and its dependencies.

03 BuildCreate positives, negatives, repairs, preferences, and variants.

04 VerifyAttach deterministic, Lean, simulator, or reviewer outcomes.

05 Re-runMeasure improvement on sealed holdouts and refreshed tasks.

Dataset catalogue and technical details

Verification and intended use determine whether an asset belongs in training. Derived task families are clean-room environments inspired by a benchmark's capability surface, not copies of its original questions.

Compare products, verification boundaries, and access

Asset	Best for	Verification boundary	Access
Math RL Research-level tasks at three difficulties: ultra-hard, hard, and medium	Research-math RL training, conjecture exploration, verifier-backed capability evaluation, and advanced agent curricula	Task-specific exact, proof, witness, construction, structured-output, or certificate verifiers; explicit acceptance criteria and reference solutions	Private suites · View sample
Terminal-Bench Science-derived Scientific-agent environments	Scientific-agent RL, hidden-instance generalization, reusable solver evaluation, and mathematical science	Exact checks, certificate or witness validation, stateful simulator scoring, executable tests, and schema checks	Private Harbor-compatible environments · View sample
Terminal-Bench-derived Hard terminal-agent environments	Terminal-agent post-training and clean-room private evaluation	Deterministic or exact checks, packaged verifier code, executable tests, and structured-output checks	Private Harbor-compatible environments · View sample
Verified Research Reasoning 1,000+ trajectory program	RLVR, process supervision, critics, first-bad-step detection, proof repair	Review and machine-checkability recorded per unit; not blanket formal verification	3 public inspection records · Licensed reviewed packs and holdouts
OlympiadNet-Math 20,000+ tiered records	Olympiad SFT, proof attempts, preference data, promoted RLVR tasks	Quality tiers and promotion gates; only a strict subset is positive-weight RLVR	10 public inspection records · Tiered private corpus
ArxivNet 57,648 canonical records	Long-context math SFT, reconstruction, PRM, critics, preferences	Structurally validated candidate layer; mathematical review required	Private candidate and review-ready exports
Private Mathematics Problems 100,000+ internally created, unpublished problems	AIME and olympiad training or evaluation, graduate and PhD reasoning, RLVR, and open-research work	Exact-answer contracts where appropriate; RL tasks are math problems with executable verifiers; open problems use scoped computation and expert review	Private packs and holdouts · Public samples: AIME++, Math RL Tasks, and SOTA Math
AIME++ Exact-answer mathematics	Competition-to-hard exact-answer RLVR, scalable curricula, and sealed private evaluation	Integer normalization and exact match; boxed-answer parsing is reported separately	Private packs · View sample
AIME-Graduate Graduate mathematics with an exact-answer contract	Graduate-level RLVR with a deterministic terminal reward	Integer normalization and exact match; boxed-answer parsing is reported separately	Private packs · View sample
AIME-Researcher Research-level mathematics with an exact-answer contract	Research-level exact-answer RLVR and frontier difficulty profiling	Integer normalization and exact match; boxed-answer parsing is reported separately	Private packs · View sample
CritPt-derived Physics RL environments	Benchmark-targeted post-training and clean-room private evaluation	Deterministic or exact checks, packaged verifier code, executable tests, and schema checks	Private Python environments · View sample
GDPVal-derived Synthetic professional-work RL environments	Long-horizon professional agents, artifact production, quantitative reasoning, and cross-file consistency	Deterministic dense rewards across correctness, instruction following, traceability, artifact integrity, consistency, quality, and safety	Private Harbor-compatible environments · View sample
GDP.pdf-derived Long-horizon multi-document agent RL	Retrieval, rule reconciliation, quantitative reasoning, evidence attribution, optimization, and structured decisions	Deterministic dense and sparse grading requires correct values, physical-page or source-record evidence, and passing prerequisite chains	Private DocumentEnv delivery · View sample
GPQA Diamond-derived Graduate-science RL environments	Benchmark-targeted post-training, clean-room evaluation, and repeatable science-reasoning rewards	Deterministic or exact checks, packaged verifier code, executable tests, and schema checks	Private Harbor-compatible environments · View sample
Humanity's Last Exam-derived Frontier-reasoning environments	Benchmark-targeted post-training, clean-room evaluation, and difficult reasoning curricula	Deterministic or exact checks, packaged verifier code, executable tests, and schema checks	Private Harbor-compatible environments · View sample
Agents' Last Exam-derived Multi-mission planning and data-processing environments	Long-horizon agents, optimization, and cross-domain workflows.	Executable checks, artifact validation, and objective verification.	Private Docker environments · View sample
SWE-Bench-derived Compact Python systems-software repair	Software-repair RL and hidden-test generalization.	Public regression checks and private executable tests.	Private Docker environments · View sample
DeepSWE Systems-software repair across four languages	Software-repair RL, recovery semantics, and hidden-test generalization.	Compiled test suites and isolated Python test workers.	Private Docker environments · View sample
SciCode-derived Scientific coding RL environments	Scientific coding-agent RL and weighted numerical evaluation	Weighted numerical checks, executable tests, and structured-output checks	Private Python environments · View sample
SciDiamond-derived Synthetic advanced-science RL	Advanced-science RLVR curricula and locked-test evaluation	Deterministic exact four-option rewards, packaged verifier code, and executable tests	Private Harbor-compatible environments · View sample
Tau-Banking-derived Stateful banking-agent environments	Stateful tool-use RL, banking-policy compliance, and database-state evaluation	Deterministic outcome and compliance checks, packaged verifier code, and executable tests	Private Python environments · View sample
CyberSec-defensive Synthetic cybersecurity repair environments	Defensive coding-agent RL and local incident or security-system repair	Deterministic partial-credit checks, packaged verifier code, and structured-output checks	Private Python environments · View sample
CyberSec-def+offense Offline cybersecurity labs and defensive-agent simulations	Cybersecurity-agent training, RL, and private evaluation.	Deterministic verifiers, action replay, hidden tests, and artifact checks.	Private Docker environments · View sample
AA-LCR-derived Long-context reasoning environments	Long-context structured reasoning, precise extraction, and private-gold evaluation	Deterministic JSON verification with shaped field rewards and strict pass/fail	Private Python environments · View sample
AA-Omni-derived Factual-reliability and abstention environments	Factual-reliability RL, calibration, and abstention behavior	Deterministic verification with correct, incorrect, partial, and not-attempted outcomes	Private Python environments · View sample
AA-Briefcase-derived Evidence-driven business analysis and planning	Evidence-grounded computational-analysis RL, optimization, accounting and reconciliation, operational decision-making, structured artifact generation, and deterministic verifier integration.	Deterministic contract-based terminal grading across numeric/accounting correctness, feasibility, optimality, evidence, memo compliance, and later certificate validity; infeasible optimization plans can receive zero reward.	Private Docker environments · View sample
AutoBench Stateful enterprise workflow simulations	Business-process automation, tool use, and policy compliance.	Deterministic action replay and state/policy checks.	Private Docker environments · View sample
Toolathlon Multi-tool operational workflows	Tool orchestration, long-horizon planning, and workflow RL.	Deterministic action replay and private outcome checks.	Private Docker environments · View sample
ErdősBench Research reasoning benchmark	Research behavior, proof gaps, counterexamples, calibrated progress	Judge and proof audit; not formal theorem certification	Public smoke test · Private full benchmark
SimoBench Olympiad proof benchmark	Olympiad proof construction, partial credit, false-solve analysis	0–7 mathematical grading against references	Public research benchmark · Private refreshes
Custom data packs	Model-specific weaknesses and private regression suites	Verifier contract designed around the target task	Private engagement

Training views: SFT, DPO, critic and repair data, RLVR/GRPO tasks, process-reward steps, private evaluation, and regression tests.

RL tasks: math problems with executable verifiers, explicit answer contracts, reward rules, adversarial checks, and private train/development/holdout variants.

Environment delivery: task families can use Harbor-compatible containers, Python agent environments, stateful simulators, or custom Gym-style APIs while private manifests and answer keys remain protected.

Start with a dataset you can inspect.

Review Verified Reasoning, OlympiadNet, AIME++, Math RL Tasks, SOTA Math, and the benchmark-derived task samples—or discuss private mathematics problems, ArxivNet, larger reviewed packs, sealed holdouts, and custom data built around your model's actual failures.

Discuss data access Browse public datasets

Find where reasoning fails.Train the fix.