One of the more unsettling findings in recent AI safety research is that misalignment generalises. Train a model to write insecure code, and it starts giving harmful advice, behaving deceptively, or sabotaging safety evaluations across entirely unrelated domains. This phenomenon, known as emergent misalignment, suggests that narrow misbehaviour training selects for something deeper than task-specific bad habits. It appears to shift the model's underlying behavioural disposition.
The natural follow-up question is whether the same mechanism can work in reverse. Can reinforcement learning on explicitly beneficial behaviour produce alignment that generalises broadly, rather than alignment that is fragile and domain-specific? A new paper from OpenAI researchers, Reinforcement Learning Towards Broadly and Persistently Beneficial Models, provides substantial empirical evidence that the answer is yes.
What the Paper Does
The authors frame their work around three distinct failure modes for deployed AI systems: failure to generalise aligned behaviour beyond training contexts, acquisition of misaligned strategies during RL itself (reward hacking, deception, specification gaming), and vulnerability to adversarial steering after deployment. These are not hypothetical concerns. They are documented failure modes in current systems.
Their approach involves constructing a multi-domain dataset targeting specific beneficial traits: truthfulness, fairness, risk awareness, corrigibility, metacognitive transparency, and downside-aware planning, among others. These traits are operationalised across realistic conversational scenarios spanning health, law, science, education, and business. Models are then trained with RL on this dataset and evaluated against more than 50 independently constructed alignment and benefit benchmarks that were not part of training.
The three primary contributions are:
- A multi-domain beneficial trait dataset suitable for both evaluation and RL training
- Evidence that beneficial trait RL produces broad out-of-distribution alignment generalisation
- Evidence that this training improves alignment persistence under adversarial prompting and harmful fine-tuning
Methodology and Experimental Design
The experimental design is careful in ways worth highlighting. Rather than simply training on a broad beneficial dataset and comparing to an untrained baseline, the authors use a compute-matched baseline throughout most of their analysis. This controls for the possibility that any RL training, regardless of content, might produce alignment improvements simply through additional compute or training steps.
The two most important ablations are the domain transfer controls. In one experiment, the beneficial trait intervention is restricted entirely to health-domain conversations, representing only 5% of total training compute. The remaining 95% of training is identical between the two models. Despite this narrow intervention, the health-only model improves performance on non-health alignment benchmarks, including reward hacking in code, chain-of-thought deception evaluations, and general misalignment measures. In the complementary experiment, health and science are explicitly excluded from the beneficial trait data allocation, yet the resulting model still improves on health and mental health evaluations. These two controls together make a direct domain overlap explanation implausible.
The paper also addresses the evaluation awareness concern directly. If models simply learn to recognise evaluation contexts and perform better in them without genuine behavioural change, the results would be misleading. The authors report that 16 of their 53 evaluations use privacy-preserving production traffic data rather than synthetic benchmarks. On this production-data subset, beneficial trait RL outperforms the baseline on 14 of 16 evaluations with a mean improvement of 3.6 percentage points. This does not eliminate evaluation awareness as a partial contributor, but it does undercut a purely artifactual explanation.
Results
The headline numbers are strong. Across all 53 out-of-distribution evaluations, the beneficial trait model outperforms the compute-matched baseline on 44 (83%), with a mean improvement of 9.1 percentage points. After Benjamini-Hochberg false discovery rate correction, 30 of these improvements remain statistically significant. Only 3 evaluations show significant regression.
Specific results worth noting:
- Substantial gains on deception benchmarks (DeceptionBench, MASK) and reward hacking benchmarks (School of Reward Hacks, EvilGenie variant)
- Improvements on broader alignment benchmarks including PropensityBench, Machiavelli, and AgentHarm
- On 10 health and mental health evaluations, the model improves on 9 (90%), with mental health assistance scores improving from 0.385 to 0.479 on one evaluation and from 0.463 to 0.519 on another
- On alignment persistence under harmful fine-tuning: after training models to produce bad medical advice, the beneficial trait RL model shows substantially less degradation on broader alignment measures (Misalignment falls by 0.08 versus 0.36 for the pre-RL baseline; Alignment Questions falls by 0.07 versus 0.46)
The paper also reports a preliminary analysis of correlation structure across alignment evaluations on OpenAI models from o3 to GPT-5.5. Alignment evaluations are weakly but significantly correlated across models (mean Spearman rho = 0.107 against a null interval of [-0.019, 0.029]), and the first principal component explains 28.2% of variance, well outside the null interval of [15.3%, 20.8%]. This is consistent with the hypothesis that diverse alignment evaluations partly reflect shared underlying behavioural tendencies rather than entirely independent skills, which motivates the whole approach.
Limitations and Open Questions
The authors are reasonably candid about what remains uncertain. The harmful fine-tuning persistence comparison uses a pre-RL baseline rather than the compute-matched standard RL baseline used elsewhere, which means it does not isolate whether persistence is specific to beneficial trait RL or whether high-compute RL more generally entrenches alignment-relevant behaviours. This is a real limitation and the authors flag it explicitly.
The paper also does not fully resolve the mechanism. The persona selection framework (Marks et al., 2026) provides a plausible account, where post-training shapes a high-level assistant persona whose traits generalise across domains, but the mechanistic evidence for this in the current paper is correlational rather than causal. Understanding precisely what changes in the model's representations during beneficial trait RL, and why those changes transfer, remains open.
There is also a normative question the paper appropriately does not try to settle. The specific traits chosen as beneficial reflect particular value judgements. The authors acknowledge that determining which values advanced AI systems should embody is a broader question requiring societal deliberation, not just empirical optimisation. This is the right framing, but it means the approach as described is not fully specified without prior agreement on what counts as beneficial.
What the paper does establish convincingly is that RL need not be a net source of alignment risk. The same exploratory dynamics that can amplify reward hacking can, when the reward signal is well-targeted, reinforce behavioural priors that generalise constructively. That is a meaningful empirical contribution to a field that has often treated RL primarily as an alignment hazard to be managed rather than a tool to be directed.
For researchers working on AI safety, scalable oversight, or post-training methodology, this paper is worth reading in full: arxiv.org/abs/2606.24014.