// Verifier-first research agent environment
ExpGraph evaluates research behavior, not polished explanations. A model must adapt under distribution shift, choose informative interventions, repair a mechanism, manage uncertainty, and submit evidence that survives sealed verification.
// 01
Most evaluations ask for a final answer. ExpGraph asks whether a model can make a sequence of research decisions that improves a hidden outcome—and whether the resulting claim is supported by an auditable chain of evidence.
THE UNIT OF EVALUATION IS
A VERIFIED RESEARCH LOOP.
Useful for measuring local task performance. It says less about whether the model can choose information, spend experimental budget, stop at the right time, and transfer beyond what it saw.
The evaluator scores hidden transfer, mechanism support, uncertainty, efficiency, parsimony, and experiment choice. The result is bound to the exact pack, schema, transcript, and submission digest.
// 02
ExpGraph exposes a persistent, bounded decision process. The policy must decide what the public evidence implies, what information is missing, which action is worth its cost, and when the evidence is strong enough to submit.
OBSERVE → HYPOTHESIZE → INTERVENE
→ REPAIR → VERIFY.
Stage 01 / public evidence
The observation contains useful evidence and an explicit contract—but omits the hidden family, generator seed, true mechanism, replay batches, and evaluator-only diagnostics.
Stage 02 / method selection
The policy must decide whether the current observations support a global correction, a prior shift, a missing interaction, a leakage term, or no intervention at all.
Stage 03 / active experiment
MechanismRepair accepts bounded initial states, observation times, controls, and replicates. Physical designs are canonicalized so changing IDs or prose cannot disguise a duplicate intervention.
Stage 04 / artifact construction
The final artifact is bounded and machine-checkable: term IDs, coefficients, uncertainty intervals, limitations, method metadata, and optional provenance graph.
Stage 05 / sealed outcome
ShiftCal replays the submitted method across hidden batches. MechanismRepair evaluates unseen interventions and multiple grounded components. The lab receives a trainer-safe report and signed receipt.
// 03
ShiftCal tests method selection under distribution shift. MechanismRepair tests whether an agent can use active experimentation to recover and validate a better dynamical model.
20 SHIFTCAL CASES.
16 MECHANISMREPAIR CASES.
Select and parameterize a bounded calibration or adaptation method from labeled source data and an unlabeled target batch—then transfer that method across three sealed replay batches without breaking utility.
Inspect public trajectories and a nominal dynamical model, spend a bounded experiment budget on interventions, then submit a parsimonious symbolic repair with uncertainty and limitations.
// 04
ExpGraph links observations, hypotheses, interventions, measurements, model artifacts, claims, and receipts. The structure makes provenance inspectable while keeping truth scoring inside the task evaluator.
CLAIMS DO NOT SCORE THEMSELVES.
V0.3 GRAPH WEIGHT = 0.
// Interactive provenance graph
Select a node// 05
ExpGraph returns a versioned score plus grounded components, cost, validity, and safe failure categories. Comparable results must share the same evaluator, pack digest, reward schema, and receipt key.
REPORT THE COMPONENTS.
BIND THE SCORE TO ITS CONTEXT.
EG-001 / sealed replay objective
Each batch combines normalized NLL, Brier score, calibration divergence, and expected calibration error relative to evaluator references. A utility gate invalidates methods that damage top-1 accuracy beyond tolerance.
EG-002 / grounded component weights
// 06
The external pilot is a submission-only evaluation boundary. The lab runs its own model, weights, tools, and inference stack; ExpGraph receives bounded JSON actions and returns serialized observations, trainer-safe results, and signed receipts.
NO CUSTOMER CODE EXECUTION.
NO ARBITRARY FILE UPLOAD.
// Data-only evaluation boundary
Lab-operated policy// 07
ExpGraph v0.3 deliberately avoids hostile-code execution. The smaller surface lets the pilot focus on benchmark secrecy, tenant isolation, durable accounting, strict input handling, and independently verifiable outcomes.
CONTROLLED DESIGN-PARTNER PILOT.
NOT PUBLIC MULTI-TENANT SAAS.
Bounded JSON actions only. No source code, model weights, arbitrary files, customer datasets, or imported lab modules.
Rejects duplicate keys, NaN, infinities, oversized bodies, excessive depth, unknown fields, and invalid numerics.
AES-256-GCM case envelopes with authenticated metadata; signed manifests list case digests and validity windows.
The offline benchmark-pack signing key is kept separate from the online evaluator and receipt-signing runtime.
Scoped credentials, allowed tasks, active-session limits, quotas, opaque cases, and one attempt per hidden case by default.
Request-body-bound idempotency prevents ambiguous retries from consuming another case, action, or experiment budget.
Policy, trainer, and audit views keep hidden per-case diagnostics away from live agents and standard lab reports.
Labs can verify receipts against retained reports, submissions, transcripts, pack identity, and a pinned public key.
// 08
Freeze the model and inference policy, run a paired or single-model campaign against one sealed pack, account for every allocated case, and close with verified results plus a concrete map of research strengths and failures.
ONE TENANT PER MODEL OR CHECKPOINT.
ONE FROZEN PACK PER CAMPAIGN.
Agree model identity, assistance rules, inference budget, task coverage, quotas, invalid-run treatment, publication rights, and analysis plan.
Test authentication, catalog access, one smoke session per task, idempotent retry, result export, and local receipt verification.
Freeze evaluator, pack, reward schemas, and receipt key. Every created session counts, including invalid, abandoned, and expired attempts.
Verify receipts, aggregate only compatible results, report coverage and component profiles, revoke credentials, and record data disposition.
// 09
The pilot does not end with a leaderboard number. It delivers the protocol identity, coverage accounting, signed outcomes, component profiles, and a clear statement of what the synthetic environments do—and do not—establish.
A VALID NEGATIVE RESULT
IS STILL A COMPLETED EVALUATION.
Trainer-safe score, validity, cost, grounded components, receipt, and exact evaluator/pack/schema identity.
Receipt-verified count, mean, median, range, variation, coverage, and compatible-result checks.
Shift transfer, experiment choice, interventional fit, mechanism support, uncertainty, efficiency, and parsimony.
Public observations, accepted actions, measurements, remaining budget, status, and immutable session history.
Invalid actions, utility failures, model-form errors, weak experiments, transfer gaps, and stop/continue mistakes.
What the evidence supports, what it cannot prove, and the highest-value next environment or training-data intervention.
Customer-safe output
The report leads with the measured question, exact protocol identity, inclusive coverage, validity, component scores, cost, and limitations. Hidden labels, mechanisms, replay batches, and operator-only diagnostics remain excluded.
// 10
The protocol is explicit about what is measured, who controls each boundary, and where the current pilot stops.
SELL IT AS VERIFIED RESEARCH-SKILL
EVALUATION—NOT AUTONOMOUS SCIENCE.
Both boundaries exist, but they serve different purposes. Trusted internal Python environments can support high-volume training. The external v0.3 pilot is a frozen, one-attempt-per-case evaluation surface and should not be repeatedly probed for training.
No. The standard external pilot accepts bounded JSON actions only. The lab keeps model code, weights, tools, and inference infrastructure in its own environment. ExpGraph does not accept arbitrary file uploads or customer code execution.
Whether a model can select a reusable calibration or shift-adaptation method from public evidence and transfer it across one visible and three sealed target batches while preserving utility. The score is shaped benchmark reward, not a production safety guarantee.
MechanismRepair measures how strongly accepted interventions distinguish the hidden truth from the nominal baseline per credit, normalized against a sealed generated candidate design set. It rewards useful information—not experiment count—and does not claim globally optimal experimental design.
No. In v0.3, it is an optional structural provenance object with zero reward weight. Free-form claims, graph size, prose length, and citation padding cannot improve the task score. Future semantic claim verification would require separately versioned predicates.
No. The launch tasks are synthetic, procedural research-skill environments. They measure bounded capabilities under declared generators and verifiers. Domain transfer, scientific validity, and deployment safety require additional real-domain evidence.
Yes, as a separately scoped environment and a separately versioned benchmark pack and reward schema. The strongest expansion path is to preserve the same sealed-action, executable-verification, disclosure-tier, and signed-evidence architecture.
// ExpGraph pilot
Ulam will freeze the protocol, operate the sealed evaluator, verify every result, and turn the failure profile into the next environment or training-data opportunity.