Private AI Evaluation Services: Measuring What Public Benchmarks Miss

AI evaluation has a visibility problem.

The easiest results to compare are usually public leaderboard scores. They are useful: shared benchmarks create common reference points, make model generations comparable, and help the field track broad progress.

But the questions facing a model team are rarely as broad as “How intelligent is this model?” They are narrower and more operational.

Can the model distinguish a complete proof from a plausible sketch? Does it apply external theorems within their actual hypotheses? Does a new post-training run improve counterexample search while degrading proof completion? Is the model better because of the checkpoint, the prompt, the inference budget, or the judge? Does it continue to reason correctly when tools return an unexpected observation? Can its strongest claims survive expert review?

A public score usually cannot answer these questions. A private evaluation can.

That is the purpose of private AI evaluation services: to build a controlled measurement system around the capabilities, failure modes, and deployment decisions that matter to a specific model team.

Private evaluation is not secrecy for its own sake. It is a protected evaluation boundary.

The model may see the task statement, but it should not have access to the answer key, grader notes, hidden metadata, adversarial variants, or verifier logic. Training teams may inspect public examples and schemas, but private holdouts must stay outside the training pipeline. The benchmark version, model configuration, prompt, sampling parameters, tool permissions, and grading protocol should still be recorded precisely.

Private does not mean opaque. The tasks may be sealed; the protocol must not be.

Dimension	Public benchmark	Private evaluation
Task visibility	Statements and often solutions are public	Tasks, keys, rubrics, or variants remain restricted
Primary purpose	Broad comparison	Model selection, diagnosis, or release gating
Distribution	Fixed and shared	Custom, hidden, and refreshable
Scoring	Often one aggregate metric	Per-skill scores, failure classes, and severity
Typical output	Leaderboard position	Technical readout and improvement plan
Lifecycle	Run periodically against a fixed set	Refresh as the model and training data change

Ulam’s evaluation design keeps private tasks, rubrics, answer keys, holdouts, and verifier internals separated from the information exposed to solvers and trainers. At the same time, runs remain auditable through stable IDs, manifests, raw outputs, configuration metadata, judge cards, and score exports.

A strong public benchmark is valuable precisely because many teams use it. But that success creates pressure on the measurement.

Problem statements circulate. Solutions appear in repositories, papers, tutorials, conversations, and synthetic-data pipelines. Prompt formats become familiar. Training teams tune against reported weaknesses. Eventually, a model may perform well because it has adapted to the benchmark’s distribution rather than because it has acquired the underlying capability.

This does not make public evaluation useless. It changes what the score means.

A public score becomes evidence that a model performs well on a known test distribution. It is weaker evidence that the same model will behave reliably on new, hidden, or organization-specific tasks.

Ulam’s work on synthetic mathematics follows directly from this problem: new problem families, private metadata, held-out solutions, and refreshable evaluation batches reduce the value of simple recall and retrieval shortcuts.

A model may reach the expected conclusion through an unsupported theorem application, a missing equality case, an invalid bound, or a step that is only true under an unstated assumption.

Answer-level evaluation marks the response correct. Proof-level evaluation asks where the argument becomes valid, conditional, incomplete, or wrong.

For reasoning models, that difference is not cosmetic. A system that produces correct answers with unreliable reasoning may fail unpredictably as soon as the task distribution changes.

On a closed problem, incomplete work may deserve a low score. On a research-style problem, a new construction, a sharp special case, a decisive counterexample, or a correct reduction can be valuable even when the original problem remains open.

A binary solved/not-solved metric discards that information. The reverse error is equally dangerous: fluent partial progress can be mislabeled as a complete solution. A serious evaluation must reward valid progress while penalizing overclaiming.

Two models can receive the same total score for very different reasons.

One may be excellent at finding the central idea but poor at closing proofs. Another may be cautious and reliable but rarely discover a decisive obstruction. A third may solve routine cases while failing on exact constants, theorem scope, or tool-conditioned reasoning.

A model team cannot design the next training run from a single number. It needs a capability map and a failure taxonomy.

Private evaluation should not begin with a convenient dataset. It should begin with a decision.

Are we choosing between checkpoints? Testing a quantized or compressed model? Evaluating a new inference configuration? Validating an RLVR run? Setting a release gate? Measuring whether an agent uses tools correctly?

The decision determines the capabilities that matter. Those capabilities determine the task distribution and rubric.

For a mathematical reasoning model, the target failure modes might include:

• missing an equality or boundary case;
• using a theorem outside its stated scope;
• asserting the wrong exact constant;
• converting numerical evidence into an unsupported proof;
• failing to search for a simple counterexample;
• preserving an invalid premise across a long derivation;
• calling a partial or conditional result “solved”; and
• ignoring a verifier or tool observation that contradicts the current approach.

These are more useful than broad labels such as “weak at geometry” or “strong at number theory.” They identify behavior that can be measured, reviewed, and trained. The benchmark can then include ordinary items, adversarial variants, near-miss statements, proof-repair tasks, and examples designed to distinguish neighboring failure modes.

A private evaluation service should cover the full path from question design to training handoff.

Stage	Key question	Primary output
Scope	What decision will this evaluation support?	Capability map and success criteria
Build	Which hidden tasks separate the relevant behaviors?	Versioned benchmark and sealed holdout
Run	How will checkpoints and configurations be compared fairly?	Frozen prompts, budgets, tools, and sampling settings
Judge	What counts as correct, partial, misleading, or unsafe?	Rubrics, verifier contracts, and adjudicated grades
Diagnose	Where does the reasoning first break?	Failure taxonomy and regression analysis
Improve	Which failures can become training signals?	SFT, preference, critic, or RLVR artifacts
Refresh	Does the improvement survive on new tasks?	A new hidden round and longitudinal report

The first stage is especially important. An evaluation designed for model selection may emphasize relative ranking under a fixed inference budget. A release gate may care more about the rate of severe false claims. A post-training evaluation may need enough item-level detail to explain why one training recipe improved.

The same task suite cannot be expected to answer every question equally well.

Not every reasoning task can be graded by exact match. Not every proof needs a human expert to inspect every line.

The right evaluation uses the strongest appropriate verifier for each claim.

Some outputs can be checked symbolically. Finite computations can be tested by scripts. Code can be run against unit tests. Structured outputs can be schema-validated. Formal proof obligations can be checked by a theorem prover. Applications of external results may require theorem-scope review. Long informal proofs may need gold rubrics, judge-assisted triage, and expert adjudication.

Ulam’s Verified Research Reasoning Trajectories for RLVR work treats these verification modes explicitly rather than calling every task “machine-verifiable.” It also decomposes complex arguments into Proof Verification Units: small, dependency-aware claims that can be checked, scored, challenged, or repaired individually.

This produces more useful evaluation output than a bare scalar. A model team can receive:

• the total score;
• per-skill and per-item grades;
• the first failed proof unit;
• the failure type;
• missing dependencies and overclaim penalties;
• reviewer notes; and
• the valid prefix that should be preserved during repair.

That information is useful both for evaluation and post-training.

Mathematics combines two properties that are unusually valuable for model evaluation.

First, correctness has real structure. A construction has the required property or it does not. A counterexample refutes the statement or it does not. A theorem application satisfies its hypotheses or it does not.

Second, a mathematical result is rarely reducible to its last line. The path matters. Definitions, reductions, examples, lemmas, dependencies, exceptional cases, and proof closure all expose distinct reasoning capabilities.

Ulam uses two complementary evaluation regimes.

SimoBench and synthetic olympiad-style evaluations test whether a model can turn an idea into a complete proof on new, low-recall tasks. Responses can be graded on a 0-7 olympiad scale, preserving meaningful partial progress while distinguishing it from a finished solution.

ErdosBench and research-level evaluations measure a different profile: obstruction finding, counterexample search, theorem selection, finite experimentation, proof-gap detection, calibrated partial progress, and restraint around strong claims.

The distinction matters. A model can be strong at closed theorem proving but poor at research triage. Another can identify excellent ideas while repeatedly overclaiming that they close the problem. A useful private evaluation should reveal both.

A private evaluation should produce more than a slide containing a leaderboard.

A useful delivery includes the benchmark version and run manifest; frozen model, prompt, tool, and sampling configurations; raw model outputs; item-level grades; judge and adjudication records; aggregate and per-skill scorecards; regression analysis; false-solve and failure taxonomies; and a technical interpretation of the result.

For recurring programs, the evaluation should also track whether a capability is improving over time or merely moving around. A higher aggregate score can conceal a regression in theorem scope, calibration, proof completion, or tool use.

Ulam’s recurring benchmark programs use hidden or refreshed problem rounds, comparisons across multiple model configurations, proof-validity judging, raw JSONL outputs, judge cards, skill regressions, false-solve analysis, and regular technical readouts.

Most evaluations end with diagnosis. The most valuable evaluations begin the next training run.

Once a failure has been localized, it can be transformed into several kinds of model-improvement data:

• a supervised example showing the corrected reasoning;
• a proof-repair task that preserves the valid prefix;
• a preference pair contrasting a cautious partial result with an overclaimed proof;
• a critic prompt asking for the first invalid step;
• an adversarial theorem-scope variant;
• a process-reward example with step-level labels;
• an RLVR task with a verifier-backed reward; and
• a new held-out variant for the next evaluation round.

Evaluation without a training handoff is a postmortem. Evaluation with structured failure artifacts becomes an optimization loop.

This is the central Ulam workflow:

1. Build data around a target failure mode.
2. Measure the model on hidden tasks.
3. Localize the first important error.
4. Convert the error into trainable objects.
5. Serve verifier-backed tasks through the next post-training cycle.
6. Evaluate again on refreshed holdouts.

UlamGym extends this loop by exposing prompts, rewards, reward vectors, and redacted trainer replays while keeping hidden manifests and private rubrics server-side.

A private program is particularly useful when a team is:

• choosing among checkpoints, model families, or inference configurations;
• validating a fine-tune, SFT run, reward model, or RLVR intervention;
• setting a capability or reliability gate before release;
• evaluating a compressed, quantized, or distilled model;
• testing an agent that must reason over tool observations;
• investigating a regression that public benchmarks do not explain;
• building training data around a model’s observed weaknesses; or
• making external capability claims that require evidence beyond self-reported solves.

The common factor is that the decision is important enough that a familiar aggregate score is insufficient.

Private evaluation can still be done badly.

A sealed benchmark is not automatically rigorous. It can be unrepresentative, inconsistently graded, too small, or quietly tuned against. A judge model can introduce its own preferences and blind spots. A fixed private set can eventually become an internal training target. A team can cherry-pick the configuration, sampling budget, or metric that produces the best headline.

The safeguards are straightforward but demanding: version the suite, freeze the protocol before running, separate training and holdout access, record all relevant configurations, audit judge reliability, retain raw outputs, adjudicate important disagreements, report limitations, and refresh the task distribution.

A private score should be harder to game - not harder to inspect.

The frontier is not short of benchmark numbers.

What model teams need is a trustworthy answer to a more useful set of questions: What can this specific model do under these constraints? Where does its reasoning break? Which strong claims survive review? What changed between checkpoints? Which failures are severe enough to block release? Which examples, critiques, rewards, and adversarial tasks should be created next?

Private AI evaluation services should answer those questions with hidden tasks, reproducible protocols, reasoning-level grading, and a direct path from evaluation failure to model improvement.

Ulam builds private benchmark programs for research-level and olympiad-level mathematics, custom evaluations targeted to model-specific weaknesses, training-ready reasoning data, and verifier-backed RLVR environments.

Measure the reasoning. Find the first real failure. Train on what the benchmark reveals.

Talk to Ulam about a private evaluation →