Full-pass rate against mean reward
Mean reward can never fall below the full-pass rate. The height above the diagonal is the average partial credit earned per graded problem.
Ulam / Security reasoning evaluation
CyberBench.
Security reasoning.
Verified in execution.
A challenging evaluation of AI security reasoning, measured through isolated execution and native private verification. Compare complete solutions, partial credit, scoring coverage, and refusals with the run settings in view.
60problems · 4 capability groups
50 / 10defensive / offensive
10runs · 600 problem attempts
28.57%highest full-pass rate · GPT-6.1 Sol
01 / Model results
The results, with their run settings.
Higher full-pass rates and mean rewards indicate better measured performance.
CyberBench · 60 problemsPrivate verification · aggregate results only
| # | Model / setting | Full-pass rate graded outcomes | Mean reward graded outcomes | Numeric coverage | Outcome distribution full / partial / zero / unscored | Refusals all 60 | Description |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6.1 SolCodex | 28.57%16 / 56 graded | 0.7138reward / 1 | 56 / 604 unscored | 16 / 36 / 4 / 4 | 4 / 606.67% | Broad defensive progress across identity, protocol integrity and incident response, with many submissions close to full reward. The recurring gap is turning strong partial results into fully verified solutions. Refusals limit its adversarial coverage. |
| 2 | DeepSeek V4.1 Flash 1017DeepSeek · API | 13.33%8 / 60 graded | 0.3526reward / 1 | 60 / 600 unscored | 8 / 24 / 28 / 0 | 0 / 600.00% | Combines five adversarial passes with three defensive passes across protocol integrity and incident response. Broad partial progress in identity and protocol work includes many near-complete submissions, although most defensive problems still fall short of a full solution. |
| 3 | GLM 5.3 FlashZ.ai · API | 11.67%7 / 60 graded | 0.1434reward / 1 | 60 / 600 unscored | 7 / 10 / 43 / 0 | 0 / 600.00% | Six of its seven full passes come from adversarial reasoning, giving this run a concentrated strength in constructing successful attack artifacts. Defensive results are much less consistent, with only one protocol-integrity full pass and no incident-response credit. |
| 4 | Gemini 4 ArgonGoogle · API | 10.00%6 / 60 graded | 0.1956reward / 1 | 60 / 600 unscored | 6 / 12 / 42 / 0 | 0 / 600.00% | Four adversarial passes are joined by complete solutions in identity and protocol integrity. Several other defensive submissions approach full reward, but incident response earns no credit. The run shows strong individual defensive results alongside many zero-score outcomes. |
| 5 | HY4 PreviewTencent · API | 10.00%6 / 60 graded | 0.1721reward / 1 | 60 / 600 unscored | 6 / 14 / 40 / 0 | 0 / 600.00% | Broad partial progress in identity and protocol integrity, including some nearly complete defensive results. Most full solutions still come from adversarial reasoning. Incident-response credit is minimal, so useful intermediate work does not yet become reliable completion across the benchmark. |
| 6 | DeepSeek V4.1 FlashDeepSeek · API | 8.33%5 / 60 graded | 0.1131reward / 1 | 60 / 600 unscored | 5 / 10 / 45 / 0 | 0 / 600.00% | Complete solutions concentrate in adversarial reasoning, with one additional identity success. Protocol work mostly earns small partial rewards, and incident response earns none. This run produces isolated complete solutions without similarly broad defensive coverage. |
| 7 | Muse Wonder 1.0Meta · API | 8.33%5 / 60 graded | 0.0928reward / 1 | 60 / 600 unscored | 5 / 9 / 46 / 0 | 0 / 600.00% | All five full passes come from adversarial reasoning. Defensive progress is sparse: nine submissions receive small partial rewards, spread across identity, protocol integrity and incident response, with no defensive full passes. |
| 8 | Mistral Large 4Mistral · API | 6.67%4 / 60 graded | 0.0895reward / 1 | 60 / 600 unscored | 4 / 10 / 46 / 0 | 0 / 600.00% | Occasional near-complete protocol work, but little consistent defensive credit. All four full passes come from adversarial reasoning; identity progress is minimal and incident-response outputs earn no reward. Strong individual results do not translate into broad defensive performance. |
| 9 | MiMo V2.6 ProXiaomi · API | 5.00%3 / 60 graded | 0.0574reward / 1 | 60 / 600 unscored | 3 / 8 / 49 / 0 | 0 / 600.00% | Three complete adversarial solutions, but very little rewarded work elsewhere. Defensive partial scores are uniformly small, with no defensive full passes or incident-response credit. Its successes are narrow rather than evidence of sustained performance across the benchmark. |
| 10 | MiniMax M3.1 FlashMiniMax · API | 3.33%2 / 60 graded | 0.0407reward / 1 | 60 / 600 unscored | 2 / 8 / 50 / 0 | 0 / 600.00% | Both full passes come from adversarial reasoning. Eight defensive submissions earn small partial rewards in identity and protocol integrity, with none reaching full completion. Incident response earns no credit; the run records no refusals. |
Full · reward = 1Partial · 0 < reward < 1Zero · reward = 0Unscored · unknown
Full-pass rate = full passes ÷ numeric grades. Mean reward uses the same graded denominator on a 0–1 scale. Refusal rates use all 60 problems. Each outcome grid has one square per problem (60 in total), sorted by outcome rather than by problem order, and includes unscored outcomes. A refusal is never silently converted into a score of zero.
02 / Reading the scores
Complete solutions and partial progress.
Both views are derived from the aggregate results above; no per-problem data is used.
Where the reward comes from
Total reward points across all 60 problems, split into full passes (exactly 1 each) and partial credit. The remainder was not earned on graded problems or was unscored.
03 / Task groups
60 cybersecurity problems.
From enforcing trust to demonstrating failures in synthetic protections.
50 defensive · 10 offensiveProblem statements stay private
Expiry & revocation
Defensive18 problems
Identity & authorization
AuthenticateAuthorizeDecide
What the model needs to do
Authenticate evidence, compose trust and access rules, and make correct authorization decisions. Handle expiry, revocation and changing identity state while preserving legitimate access.
Replay tracking
Defensive24 problems
Protocol & system integrity
ValidateTrack stateEnforce
What the model needs to do
Implement secure protocol and runtime behavior. Validate messages and software evidence, track replay and key changes, and enforce isolation or policy constraints while preserving allowed operations.
Clock disagreement
Defensive8 problems
Incident response & evidence
CollectReconcileRecover
What the model needs to do
Reconstruct what happened from authenticated but potentially inconsistent evidence. Resolve forks and clock disagreements, identify affected state, and produce a coherent audit or recovery result.
Distributed fault
Offensive10 problems
Adversarial reasoning
AnalyzeConstructVerify
What the model needs to do
Find failures in synthetic cryptographic or distributed-system constructions. Recover hidden relationships or produce a verifiable artifact demonstrating that a stated protection can be broken.
04 / Scoring & verification
One submission. Private verification.
Execution evidence determines the score. Coverage and limitations stay visible.
Independent agent evaluations.
Each problem is assigned to a fresh agent without inherited conversation history. The agent reads public instructions, works in an isolated environment, runs its own checks, and submits a frozen artifact for private grading.
An agent works on one problem at a time within the configured execution budget. Wall-clock limits range from 30 to 120 minutes per problem. Private verifier feedback is withheld during the run.
100,000 tokens per problem.
Each problem starts with a fresh conversation and tools for workspace inspection, code execution, and submission. The evaluation uses a cumulative 100,000-token input-plus-output limit per problem, alongside its configured wall-clock limit.
Reported reasoning tokens count once as part of output usage. Repeated input context is counted on every request. When usage is unavailable, the budget is reserved conservatively. The limit covers total API usage, rather than output alone.
1
Independent attempt
Fresh solver, public instructions and permitted files in an isolated copy.
2
Public checks
Offline Docker execution as UID 1000, with no Docker socket and no reference solutions.
3
Frozen artifact
The submission is fixed before grading. No verifier feedback reaches the solver.
4
Native private grade
Reward between 0 and 1. Refusals and infrastructure failures are recorded separately.
Complete solutions and partial credit.
Each successfully graded submission receives the native verifier’s reward between 0 and 1. A full pass requires reward exactly 1. Partial rewards show measured progress within the benchmark’s contract; they are not a percentage of real-world security capability.
- Full-pass rate
- Count of rewards exactly 1, divided by the number of numeric grades.
- Mean reward
- Arithmetic mean of all numeric rewards, including zeros.
- Refusal rate
- Recorded content refusals divided by all 60 problems.
Keep solving separate from grading.
Solvers work from public instructions and permitted files in an isolated copy. Candidate execution uses offline Docker containers as UID 1000, without private verifiers, reference solutions, or a Docker socket. The final artifact is frozen before a separate private grading stage.
Private verifier feedback is not supplied to the solver for repairs. Recorded content restrictions end the attempt and are saved as refusals. Infrastructure and logging failures remain unscored.
Independent attempt → public checks → frozen artifact → native private grade
Evaluate your security reasoning system.
Work with Ulam on controlled model comparisons, reproducible evaluation settings, and evidence that helps improve your agents.