// Autonomous-teacher environment
DistillSWE is a closed-loop meta-RL environment. An outer model receives a frozen coding student, visible repositories, a controlled post-training stack, and a fixed resource budget. It must create the learning loop—not merely solve a task.
// 01
Most software benchmarks measure whether a model can produce one correct patch. DistillSWE measures whether a model can diagnose a student, manufacture useful experiences, allocate post-training compute, and cause durable improvement.
THE OUTER MODEL IS THE AGENT.
THE STUDENT IS THE SYSTEM IT MUST IMPROVE.
Useful for measuring local coding ability. It says less about whether a model can run a long-horizon scientific improvement process.
The unit of success is not a clever synthetic task. It is measurable improvement of the student on private, unseen repositories without unacceptable retention loss.
// 02
DistillSWE exposes a persistent macro-action environment. The outer policy decides what to inspect, what to generate, which attempts to repair, which data to train on, and when to spend scarce development queries.
RESET → OBSERVE → STEP
EVERY ACTION RETURNS AN IMMUTABLE RECEIPT.
Stage 01 / task creation
The outer model proposes repository issues, behavioral tests, reference patches, regression commands, skill tags, and difficulty metadata. Proposals enter a staged validation pipeline before they can influence training.
Stage 02 / student inference
The same fixed coding-agent scaffold is run against accepted tasks. Every rollout records the exact base model, adapter, scaffold, decoding configuration, tool trace, patch, verifier result, and resource use.
Stage 03 / judgment and repair
The outer model may diagnose, rank, or repair student attempts. Correctness remains grounded in an organizer-owned harness outside the candidate repository, with expected-test attestation and anti-bypass checks.
Stage 04 / adaptation
The outer policy constructs datasets and spends its compute budget through a pinned LoRA stack. v0.4.1 supports supervised trajectory training, verifier-ranked preference optimization, and patch-level verifier-guided GRPO.
Stage 05 / private transfer
The final checkpoint is evaluated outside the participant action space on hidden, repository-disjoint tasks. Reward accounts for average transfer, weak-tail performance, run variance, retention, resource use, and integrity.
// 03
Labs can start with high-volume sub-environments or run the full ten-hour autonomous teaching campaign. The same repositories, verifiers, receipts, and evaluator boundaries support every track.
FORGE + CRITIC FOR VOLUME.
FULL FOR FRONTIER EVALUATION.
Selected track
DistillSWE-FullThe flagship environment. Terminal reward is sealed student transfer, adjusted for robustness, retention, compute, and integrity.
// 04
The evaluated model or policy receives a typed action interface. Ulam hosts the student, training stack, sandboxes, private evaluators, evidence service, and organizer controls in a disposable single-tenant deployment.
MINIMAX OR ANOTHER LAB CAN SUPPLY
A POLICY CONTAINER OR PINNED MODEL ENDPOINT.
// Single-tenant pilot boundary
Lab-operated model// 05
Generating more tasks, collecting more rollouts, or consuming more compute does not directly earn the terminal score. The authoritative result is whether the adapted student becomes better on a sealed distribution.
ONE STUDENT. ONE CHECKPOINT.
ONE PRECOMMITTED EVALUATION CONTRACT.
Authoritative terminal reward
Robust target performance blends aggregate capability with lower-tail skill performance. Fixed-seed lower confidence bounds discourage lucky checkpoints. Retention and compute remain explicit costs, while an integrity failure can invalidate the run.
targetP10LCBΔGcost// 06
The pilot boundary is intentionally stricter than an ordinary research harness. Participant actions are confined, transactional, schema-validated, resource-bounded, and auditable before they can affect the measured result.
CONTROLLED, DISPOSABLE,
SINGLE-TENANT PILOT RELEASE CANDIDATE.
Participant paths are resolved against action-specific roots before any read, write, deletion, or subprocess use.
Work executes in hidden staging paths and publishes atomically; rejected or interrupted actions do not commit artifacts.
Every step carries a request ID. Safe replays return the original receipt without repeating work or spending budget twice.
Candidate repositories cannot shadow or replace the verifier harness. Test counts and harness digests are attested.
Student and verifier jobs run in disposable, network-disabled boundaries with only the minimum task mounts.
Unknown fields, coercions, oversized payloads, and out-of-range resource requests are rejected.
Rollout, verifier, evaluator, and training process trees have external deadlines and forced cancellation.
Submission is adapter-only, tensor-allowlisted, shape-checked, lineage-bound, and tied to the pinned base model.
// 07
Ulam and the lab precommit the model identity, student, repositories, resource envelope, evaluator, retry rules, and success criterion before measured execution. A valid negative result is still a completed pilot.
RECOMMENDED: ONE SMOKE RUN
+ THREE INDEPENDENT MEASURED CAMPAIGNS.
Select integration mode, freeze identities, approve repository substrate, and sign the evaluation plan.
Verify API behavior, endpoint capacity, sandboxing, cancellation, evaluator schemas, and evidence generation.
Run three independent Full-track episodes from clean student checkpoints under the same pinned protocol.
Deliver per-run evidence, campaign aggregate, adapter lineage, resource ledger, interpretation, and next-step recommendation.
// 08
DistillSWE does not end at a leaderboard number. Every measured campaign produces enough protocol identity and provenance to understand what improved, under which constraints, and with which limitations.
RESULTS ARE SPECIFIC TO THE PINNED
MODEL, STUDENT, SCAFFOLD, BUDGET, AND SUITE.
Target, weak-tail, robustness, retention, compute, integrity, and terminal reward.
Three-run mean, variation, confidence interval, success rate, and comparability checks.
Submitted tensors, checkpoint ancestry, training configuration, and base-model binding.
Teacher and student tokens, training tokens, GPU time, sandbox CPU, tests, tools, storage, and wall clock.
Hashed inputs, outputs, action state, budget changes, integrity events, and provenance verification.
What the result establishes, what it does not establish, and the highest-value follow-on experiment.
Customer-safe output
The report leads with the measured question, protocol identity, raw score movement, robust reward, and capability profile. Private prompts, repositories, and hidden tests remain excluded.
// 09
The protocol is intentionally explicit about what is measured, who operates each boundary, and where the current release stops.
THE PILOT IS AN AUDITABLE MEASUREMENT.
IT IS NOT A GUARANTEED UPLIFT CONTRACT.
No. Fine-tuning is one action inside a larger environment. The outer model must diagnose the student, create and validate experiences, allocate rollouts, judge failures, construct training artifacts, choose bounded recipes, use scarce feedback, and decide when to submit.
No. Final repositories, prompts, behavioral tests, reference solutions, retention tasks, and evaluator internals remain inside an organizer-controlled sealed boundary. Development feedback is aggregate and strictly allowlisted.
For the first engagements, Ulam should host the complete environment in a disposable single-tenant deployment. The lab supplies a pinned model endpoint or policy container; Ulam operates the student, trainer, sandboxes, private suites, scoring, and evidence service.
Yes. The preferred integration is a signed policy container or a remote controller connected through an authenticated gateway. A Ulam controller calling a pinned model endpoint is also possible for a faster, narrower baseline.
Full tool-use trajectories can be trained through SFT and DPO. The bundled verifier-guided GRPO stage is patch-level. Full multi-turn student-agent policy optimization remains outside this pilot release.
The pilot still produces a valid capability result: which strategies were attempted, where the student failed to transfer, how resources were spent, and what the evidence supports. Commercial delivery acceptance is separate from a precommitted performance success criterion.
// DistillSWE pilot
Start with an integration session to select the outer-policy interface, student checkpoint, repository distribution, resource envelope, and decision criterion.