Polymorphic Reinforcement Learning: Learning When the Rules Change

An agent can become very good at the wrong problem. It can learn which action succeeds under one set of rules, then keep choosing it after those rules change. Making its behavior more consistent would not fix that failure. It would reinforce it.

This is the starting point of our paper, Polymorphic Reinforcement Learning. We study learning across executable families whose environments, rules, observations, rewards, or interfaces can change. The central challenge is conditional adaptation: reusing what remains valid while responding to what no longer does.

For language-model post-training, that creates two distinct questions: what structure can the agent reuse, and what does its training update actually improve? A valid relationship between environments answers neither question automatically. The paper connects them through information limits, policy-sharing objectives, and exact analysis of updates across variants.

A polymorphic family begins with a shared base specification and a variant descriptor. Their relationship must be executable: a generator, interpreter, transformation layer, or alignment mechanism must specify what changes and what remains fixed. A collection of unrelated benchmarks - or merely a collection of random seeds - is not enough.

The distinctions matter. Mechanism variation changes action consequences, observations, rewards, or permissions. Task-structural variation changes composed goals and dependencies. Interface variation changes low-level execution. Representation changes alter encodings and are a controlled special case, not a substitute for changing rules.

The simulator's source code does not have to change during play. A rule configuration can be part of the environment state or a descriptor sampled at the beginning of an episode. The overall simulator can remain stationary while the operative rules differ. What matters is the declared relationship between those rules and the agent's decisions.

Consider the paper's simplest counterexample. There are two equally likely variants and two actions. In one variant, only the first action earns a reward; in the other, only the second does. A policy that knows the active variant achieves full reward. Any policy forced to use the same action distribution in both variants can achieve only half of that value.

This is not an argument against shared model parameters. One network can condition on different contexts. It is an argument against forcing identical decisions when context changes their consequences. The transferable object may be a procedure for identifying and using the current rules, rather than a universally correct action sequence.

If the active rules are hidden, the agent must work from its observable history. In the paper's finite, known-family model, a posterior over both the mechanism and the current state is sufficient for control. That statement assumes the family and prior are known; it does not provide a learned world model for free.

There is also an exact limit on identifying the rules. For two equally likely variants, the best achievable identification error is one half minus one half of the greatest total-variation separation that an allowed diagnostic policy can create between their transcript distributions. If every permitted experiment produces identical evidence, the error remains one half. Rephrasing or relabeling that evidence through a transformation independent of the hidden variant cannot make the variants more statistically distinguishable.

This does not mean every agent must reconstruct every rule. Identification matters when unresolved possibilities require different decisions. Nor does it mean rephrasing cannot help a computationally limited model use information it already has. The distinction is between making evidence easier to process and acquiring evidence that was absent.

Reuse has similarly explicit conditions. Exact transport requires matching initial conditions and aligned states, actions, observations, rewards, and time units - not merely matching final answers. Under known alignment and uniformly small errors, the difference in a bounded, fixed-horizon utility is limited by the initial mismatch plus the horizon multiplied by the per-step mismatch, capped at one. Small discrepancies can accumulate over a long interaction.

Even valid transport does not make relabeled trajectories fresh, on-policy experience for the current model. Under genuine rule changes, environmental probabilities may also differ; correcting only the policy's action probabilities does not justify reusing old-rule trajectories.

The paper separates four interventions often bundled together: sampling variants, tying their decision policies, pooling reward-normalization groups, and reusing trajectories. Each changes a different part of learning.

For a one-response task, the population objective is expected reward minus a KL penalty for moving away from a reference policy. A clipped, token-averaged, or group-normalized implementation need not estimate the gradient of that objective.

For decision-preserving variants, responses can be mapped into shared semantic actions. The standard Gibbs optimum raises the weight of an action exponentially with its reward while retaining the reference policy's relative preference among equally rewarded choices. This clarifies both the opportunity and the limitation: reward can amplify successful behavior, but the unrestricted optimum does not establish what a particular neural model can represent or learn.

Now impose an additional constraint: every variant must use the same semantic action distribution. Even with identical rewards, their reference distributions may disagree. The paper derives the exact cost of this semantic tying. The shared optimum uses a weighted geometric combination of the reference distributions, and its regularized objective cannot exceed the separately optimized value.

When the semantic references coincide, tying has no such optimal-value cost. With a common reward and matching references, an executable mixture can preserve average reward while raising worst-variant reward to the former average. With differing references, however, tied mean success can move either way. A lower regularized objective is not automatically a lower raw success rate.

These results concern aligned, common-reward problems. They do not justify ignoring changed mechanisms. In interactive environments, optimization must remain causal and history-conditioned: the agent chooses actions, while the environment produces its own outcomes. Exponentially reweighting whole trajectories would generally change environmental randomness as well.

The paper's central new analysis concerns independent episodes drawn from different variants and placed in one reward-normalization group. This covers interactive rule-changing environments, not only differently worded prompts.

Fix the variants in a group before sampling, allowing repetitions. Each episode has a binary success reward. A group-relative update centers every reward using the group's average and divides it by a normalizer determined by the number of successes. All-success and all-failure groups contribute nothing to this reward term.

The exact calculation shows that the expected update is the gradient of an explicitly computable polynomial potential. The contribution of one episode depends on the other episodes' success probabilities. That potential is not generally mean success. When shared parameters make improvements on different variants conflict, changing their relative weights can change the update's direction.

The paper proves a necessary-and-sufficient condition: universal nonnegative first-order alignment with mean success holds if and only if every nontrivial success count uses the same normalizer, under the theorem's assumptions.

This is a possibility result, not a claim that every larger group harms training. The assumptions are essential: independent episodes, parameter-independent dynamics, fixed binary verifiers, differentiable positive probabilities on fixed finite supports, and scores from the policies that generated the episodes.

The result describes the reward-gradient term, not a complete KL-regularized optimizer. Active clipping, response-length weighting, adaptive within-group sampling, and dependent episodes are outside its scope. Successive trials that share adaptation memory are not independent samples; independently reset adaptation sequences are the relevant units.

Without standard-deviation normalization, a leave-one-out reward baseline recovers the mean-success gradient under these assumptions. That does not prove faster convergence: variance, curvature, and step size still determine what happens after a finite update.

The paper makes the distinction concrete with an exactly enumerated four-response calculation. Four presentations begin with success probabilities of 10%, 10%, 10%, and 50%. A shared parameter affects their probabilities in opposing directions. They use a common binary success criterion, and every sampling design spends the same four-response budget. All three use population-standard-deviation normalization and a very small stochastic step chosen to certify the direction of change.

A homogeneous group chooses one variant uniformly and samples all four responses there. An independent-variant group chooses a variant independently for each response. A fixed-quota group samples once from every variant.

The fixed-quota design produces the most mixed-reward groups and the lowest gradient variance, yet it reduces expected success. The calculation evaluates success after every possible realized update and then averages; it does not substitute the average update into the success function.

The tiny step size certifies the signs. These are probability changes in a finite-policy example, not percentage-point forecasts for language models. Their purpose is to disprove an automatic implication: more mixed rewards and less variance do not necessarily mean better learning.

Independent variant sampling has a different guarantee. Within one base problem, its expected normalized update is a positive multiple of the mean-success gradient. Across multiple bases, that multiplier can still reweight their objectives. Sampling design is part of the algorithm, not an interchangeable implementation detail.

All newly executed computations in the paper concern finite policies. Its language-model comparisons reanalyze published experiments rather than report newly trained models.

For example, the reviewed EnvHarness comparison reports ALFWorld in-distribution success rising from 81.4% to 87.9%, while out-of-distribution success changes from 89.6% to 88.8%. The comparison uses one environment seed and reports no dispersion across independent training runs, so those figures do not establish the significance of the decrease.

EnvHarness changes interaction contracts. The reviewed PrAg-PO and TA-GRPO results concern response-level augmentation, not evidence that a model learned changed transition rules.

The paper therefore offers neither a pooled PRL effect size nor a claim that environment diversity universally improves capability. It provides conditions for valid reuse and beneficial updates, together with counterexamples showing where those conditions matter.

The accompanying reproducibility package contains the manuscript source, calculation scripts, recorded results, rerun logs, provenance, and checksums. It reproduces finite-policy calculations; it does not contain an LLM training run.

Evaluation should preserve the same distinctions as the theory. Complete transformation descendants of one base specification belong in one partition. Generating many related variants does not turn them into equally many independent worlds: replication can reduce uncertainty within a base without resolving uncertainty between bases.

Budget comparisons must account for model, optimizer, verifier, memory, rollout count, and compute. More replicas can help when generating a base is expensive, but they displace independent bases when generation is cheap. Counts of environments alone do not establish sample efficiency.

The verifier must also follow the changed task. Reaching a protected endpoint does not establish that intermediate permissions were respected. A small average KL penalty on training tasks does not protect unrelated, uncovered skills; the paper's retention bound explicitly requires distributional coverage.

The practical question is consequently not just whether reward increased. It is whether the agent improved on the intended distribution of rules, under the intended information and resource constraints, without hiding failures in an aggregate.

Alongside the paper, our synthetic ARC-style demos and Astra trajectories make some of these distinctions inspectable. They are illustrative environments, not an official ARC benchmark and not training evidence for the paper.

In Counterfactual Garden, generated intervention kernels and update parameters change action consequences. In Alembic Protocol, numerical reaction parameters change the products and subsequent phase. These are mechanism-level variations under the games' declared state and action representations, rather than merely new colors or layouts.

But varying a mechanism is not the same as requiring an agent to infer it. Both games provide executable schedules, so success can come from following instructions without learning a general predictive model. Their value here is to expose that distinction, not to claim it has been solved.

The viewer supports replaying the recorded actions and trying the games through Play as a Human. The recordings are exploratory, tool-assisted interaction - not parameter post-training or a controlled estimate of PRL gains. Human practice includes reference assistance and retries.

Polymorphic reinforcement learning is not simply a prescription to generate more environments. It asks us to specify their relationships, the evidence available to the agent, and the objective induced by learning from them.

An agent should preserve a decision when the relevant semantics are preserved. It should change its decision when the rules demand it. And a training method should be judged by that intended utility - not by diversity, mixed rewards, or variance alone.

Read Polymorphic Reinforcement Learning for the proofs, download the calculation and verification package, and explore the interactive demos and trajectories.