Hillclimbing AI Benchmarks with RL Tasks

Ulam builds clean-room, verifier-backed RL environments inspired by the capabilities measured in frontier benchmarks—so teams can train against the failure surface while keeping evaluation items out of the training set.

A benchmark score is a measurement. It is not yet a training signal.

When a model fails a terminal task, a scientific workflow, a repository repair, or a multi-document business problem, the result identifies a capability gap. But a static test item rarely tells the training system how to generate more practice, how to score partial progress, or how to distinguish a useful recovery from reward hacking.

That is the opportunity behind benchmark-derived RL tasks. We decompose the behavior measured by a benchmark, build new task families that exercise the same underlying skills, and attach executable rewards. A model can then practice tool selection, state tracking, code repair, evidence attribution, scientific inference, artifact production, or calibrated abstention across many fresh environments.

We call this hillclimbing because the loop is deliberately empirical:

evaluate → identify the recurring failure → generate a clean-room task family → train with verifier feedback → re-evaluate on a sealed holdout

The target is a better model, not a memorized test set. Ulam's derived and inspired environments do not redistribute private benchmark questions or certify an official benchmark score. They provide fresh training tasks aimed at the capabilities the benchmark is trying to measure.

From one leaderboard number to a trainable failure surface

Most agent benchmarks bundle several skills into one pass rate. A terminal benchmark may require planning, shell use, dependency repair, file editing, test interpretation, and recovery from a failed command. A professional-work benchmark may require reading a source packet, reconciling rules, calculating values, producing several files, and keeping every artifact consistent.

A team cannot improve these systems efficiently by treating every failure as the same problem. We first split the benchmark-shaped workflow into observable failure modes:

  • the model chose the wrong tool or used the right tool with the wrong arguments;
  • the intermediate state was correct, but the final artifact violated the contract;
  • the answer was numerically right but unsupported by the required evidence;
  • the local patch passed one test and broke an unrelated behavior;
  • the agent failed to revise its plan after an observation contradicted it;
  • or the prompt was underdetermined and the model should have abstained.

Each failure can become a curriculum. Early tasks isolate one skill. Later tasks compose several skills under longer horizons, noisier tools, adversarial cases, and hidden state. The final training tier recreates the difficulty profile without reproducing the evaluation item.

Benchmark-inspired RL task families

The following families are available as private training or evaluation programs. “Derived” and “inspired” mean clean-room task design around a benchmark's capability surface; they do not mean that original prompts, hidden tests, private graders, or gold artifacts are included.

Task family Capability target What the RL tasks contain Sample or reference
Terminal-Bench-derived Long-horizon terminal use, system repair, and command-line problem solving Containerized tasks involving files, packages, builds, services, scripts, debugging, and hidden tests, with deterministic terminal outcomes and structured-output checks. View sample
Terminal-Bench-Science-derived Scientific agents, hidden-instance generalization, and reusable solver construction Executable inverse problems, model discovery, calibration, scientific design, and mathematical-science tasks graded through exact checks, certificates, witnesses, or stateful simulators. View sample
Terminal-Bench Science Hard Difficult numerical science, experimental design, inference, transfer, and control Scientific coding tasks and interactive observe–experiment–measure–update–control campaigns with hidden numerical packets, robustness checks, integrity gates, and protocol-aware scoring. View sample
GDPVal-derived Long-horizon professional knowledge work and artifact production Synthetic business workflows that produce spreadsheets, documents, presentations, and PDFs from reference packets, with dense rewards for correctness, traceability, artifact integrity, consistency, quality, and safety. View sample
GDP.pdf-derived Multi-document retrieval, reconciliation, calculation, and evidence attribution Document-agent environments with search, reading, bounded calculation, and submission actions. Rewards require correct values, page-level or source-record evidence, and satisfied prerequisite chains. View sample
GPQA Diamond-derived Graduate-level biology, chemistry, and physics reasoning Clean-room science problems with deterministic answer contracts, worked reasoning, executable checks, and private train, development, and evaluation variants. View sample
Humanity's Last Exam-derived Broad frontier reasoning across difficult academic and professional domains Fresh exact-answer, multiple-choice, multimodal, and multi-stage reasoning environments with packaged verifiers and sealed variants. View sample
SciCode-derived Research-style scientific programming and numerical implementation Isolated coding environments for algorithms from numerical analysis, physics, statistics, geometry, and simulation, scored with weighted numerical tests and output contracts. View sample
SciDiamond-derived Advanced-science reasoning and locked-test evaluation Clean-room biology, chemistry, and physics curricula with worked solutions, plausible distractor rationales, exact rewards, and held-out families. View sample
Tau-Banking-derived Stateful tool use and banking-policy compliance Multi-step banking workflows with scripted users, policy documents, database-changing tools, and deterministic outcome and compliance verifiers. View sample
CyberSec-derived Defensive coding agents and local security-system repair Synthetic authentication, DNS, RBAC, cloud identity, cryptographic protocol, recovery, and supply-chain repair tasks with local actions and deterministic partial credit. View sample
AA-LCR-derived Long-context extraction, reconciliation, and structured reasoning Document-heavy tasks across business, legal, scientific, insurance, finance, and incident-response settings, graded by deterministic JSON contracts and shaped field rewards. View sample
AA-Omni-derived Factual reliability, calibration, and abstention Fresh factual tasks across finance, humanities, law, medicine, science, and software, including deliberately underdetermined cases where abstention is the correct action. View sample
AA-Briefcase-derived Long-horizon agentic knowledge work Clean-room business projects built around source packets and required spreadsheets, presentations, memos, and analytical deliverables, with artifact checks and task-specific rubrics. View sample
DeepSWE-inspired Original, long-horizon software engineering Fresh repository tasks requiring substantial feature work or repair across multiple files, graded by hidden behavioral tests, regression checks, and repository integrity. Sample access on request
AutoBench-inspired Dynamic business agents, tool selection, orchestration, and recovery Generated operator roles and business workflows with APIs, distractor tools, parameter constraints, replanning, error handling, and measurable end states. Sample access on request
Agents' Last Exam-derived Real-world professional workflows with verifiable success criteria Clean-room, multi-tool assignments spanning professional domains, staged files, specialist software, and deliverables verified through artifacts, system state, or task rubrics. View sample
SWE-Bench-derived Repository issue resolution and test-driven software repair Fresh issue-style tasks in staged repositories where agents inspect code, implement patches, and satisfy target tests without breaking protected regression behavior. View sample

For the broader catalogue, including mathematics task families, visit Ulam Data.

What turns a benchmark-shaped task into an RL environment?

A prompt and a final answer are not enough. A useful environment needs an explicit action surface, observable state, a completion contract, and a verifier that rewards the intended behavior.

Executable taskThe agent receives files, tools, APIs, a repository, simulator, or document workspace—not only a question.
Measurable progressIntermediate checks expose whether the agent repaired the right subsystem, produced valid evidence, or moved the environment toward the goal.
Terminal acceptanceHidden tests, artifact checks, certificates, reference values, state comparisons, or policy rules determine whether the completed work is acceptable.

Ulam deliveries can include the environment, task manifests, answer contracts, reference artifacts, private and adversarial tests, reward components, timeout rules, model trajectories, replay logs, stable IDs, versioned splits, and evaluation reports.

The verifier boundary is always stated. Passing a numerical checker proves that the submitted object satisfies the encoded finite conditions. Passing a document grader establishes the required fields, evidence, and cross-file consistency. Neither should be advertised as proving something the verifier never checked.

How a benchmark hillclimb works

  1. Baseline the model. Run the benchmark or an agreed private analogue under a fixed harness, tool budget, context limit, and scoring protocol.
  2. Localize the losses. Separate planning failures from tool errors, state-tracking mistakes, invalid artifacts, verifier misunderstandings, brittle code, missing evidence, and overconfident answers.
  3. Generate family-disjoint practice. Build new tasks that preserve the capability demand while changing the instance, source materials, repository, parameters, and solution path.
  4. Design the reward. Combine exact terminal acceptance with carefully chosen progress checks. Test the verifier against malformed outputs, shortcuts, and reward-hacking attempts.
  5. Train and diagnose. Export tasks for RLVR or an agent runner, retain failed trajectories, and convert recurring errors into critic, repair, preference, or supervised examples.
  6. Re-evaluate on sealed tasks. Measure transfer on untouched families and, where permitted, on the original benchmark under the same protocol.

The best result is not merely a higher pass rate on practice tasks. It is a model that transfers the learned policy to new instances: choosing tools more reliably, recovering from failure, producing valid artifacts, and respecting the same constraints when the surface form changes.

Benchmark improvement without benchmark leakage

Training toward a benchmark is legitimate only if the evaluation boundary remains intact. We recommend five controls:

  • do not place official or hidden test items, gold patches, private graders, or benchmark canaries in training;
  • split by task family, repository, workflow template, or source packet—not by superficial prompt rewrites;
  • keep a rolling hosted holdout whose answers and verifier internals remain server-side;
  • version the harness, tool permissions, inference budget, and scorer so before-and-after runs are comparable;
  • report both aggregate improvement and the failure categories that remain.

Some derived families are close analogues of a public task contract; others are broader curricula around the same capability. Neither guarantees a higher official score. The defensible claim is narrower: the tasks create repeated, measurable practice on the skills the target evaluation requires, while a sealed evaluation determines whether that practice transferred.

Build the training set around the score you need to move

Teams can begin with an existing task family or commission a program around a benchmark, internal evaluation, agent harness, or recurring production failure. A pilot fixes the environment contract, reward boundary, difficulty tiers, tool surface, acceptance criteria, and holdout policy before scaling.

The resulting data can support RLVR, supervised fine-tuning, critic training, process supervision, preference optimization, failure recovery, and private evaluation. Because the tasks are executable, the same environment can produce training rewards, regression tests, and post-training scorecards.

Tell us which benchmark or internal evaluation you need to move. Ulam can turn its failure surface into clean-room RL tasks, executable verifiers, model trajectories, and sealed holdouts—then measure whether the capability actually transfers.

Explore benchmark-derived task samples or discuss a custom benchmark hillclimb.