Most evaluations of whether large language models behave ethically rely on a deceptively simple proxy: does the model agree with what humans say? If a model classifies a scenario as morally wrong and human annotators do too, that counts as evidence of alignment. A new paper from the AI Transparency Conference challenges this assumption directly, showing that agreement in final moral labels can coexist with systematic divergence in the moral reasoning behind those labels. The implications are significant for anyone who builds, evaluates, or deploys models in contexts where ethical judgment matters.
The Core Problem
The paper draws a distinction that is obvious in philosophy but underappreciated in machine learning practice: two agents can reach the same conclusion for entirely different reasons. A human annotator might judge an action wrong because it violates a specific promise; a model might judge it wrong because it causes harm. The final label matches; the moral ground does not.
This matters beyond academic tidiness. A model that tracks moral salience through a different vocabulary than humans will generalise differently to novel cases, produce explanations that do not map onto human-recognised moral categories, and may over-moralise or under-moralise situations depending on which considerations its training has foregrounded. Label-based evaluation cannot detect any of this.
The authors situate this problem within the broader alignment literature carefully. Preference learning and reinforcement learning from human feedback define alignment primarily through behavioural outputs, a lineage running from Christiano et al. (2017) through InstructGPT. Gabriel's (2020) conceptual taxonomy of alignment targets, ranging from instructions to values, provides the theoretical basis for treating output agreement as only a thin form of alignment. The paper's contribution is to test this thinness empirically rather than assert it philosophically.
Methodology
The study uses a curated 500-item benchmark derived from the ETHICS dataset, spanning five domains: commonsense morality, deontology, justice, utilitarianism, and virtue ethics. Crucially, the benchmark was filtered to include items with enough moral texture to support rationale-level annotation. Many ETHICS items are too shallow for this purpose, which is a reasonable design choice, though it means results characterise performance on a deliberately selected evaluation set rather than the original distribution.
Each item was annotated by three human annotators drawn from a pool of five, producing both a majority final label and a distribution of structured moral rationales. The model set spanned frontier systems (Claude Sonnet 4.5, Gemini 2.5 Pro, ChatGPT 5.2) and open models (Gemma 12B, Gemma 27B, and two GaMS models, Slovene-centred open models from the Gemma family).
Both humans and models were asked to return structured outputs: a final label and, depending on the domain, either a rubric-coded rationale category or a free-text explanation. The JSON schema mirrored the human annotation sheets, which is a sensible design for ensuring the comparison is between equivalent annotation objects rather than between different output formats.
The analysis operates at two levels. Label agreement is computed straightforwardly against the human majority. Rationale alignment is assessed through distributional comparison of structured rationale categories for the three rubric-coded domains: commonsense morality, deontology, and justice. Utilitarianism and virtue ethics use free-text rationales and are excluded from the quantitative rationale analysis, which is an honest acknowledgement of the method's current scope rather than a gap to criticise.
Key Findings
The results are clear and the pattern is consistent across model families:
- Frontier models achieve high agreement with human majority labels across domains. By standard benchmarking logic, this would suggest broad moral convergence.
- Rationale-level analysis reveals systematic divergence. Models more often foreground harm, respectfulness, safety, justice, and generalised moral wrongness.
- Human annotators more often invoke promissory fidelity, trust, beneficence, contextual relevance, and situated interpretation.
- The sharpest divergence appears in the deontology partition. Human annotators frequently classify weak excuses as irrelevant to a duty; models tend to classify the surrounding situation as morally wrong. The model tracks that something is ethically problematic, but loses the more precise deontological structure relating duty, excuse, and context.
- Open models show similar distributional patterns to frontier models, with some variation in the degree of divergence rather than its direction.
This pattern is not random noise. It reflects something systematic about how models trained on large text corpora represent moral categories: harm and wrongness are highly salient, while the more relational and contextually embedded categories that human moral reasoning relies on, things like trust, promise-relevance, and excuse structure, are underweighted even when they are the morally decisive considerations.
Limitations and Open Questions
The authors are candid about the study's constraints. The human annotator pool is small and shares a broadly Central European sociocultural background, so the human majority label is a useful comparison point but not a culturally representative or philosophically authoritative moral reference. This is a real limitation, not merely a boilerplate caveat: moral ground distributions will vary across communities, and a different annotator pool might shift the human baseline substantially.
The benchmark curation process, while necessary, introduces author judgement about what counts as morally substantive. The paper calls for full release of item identifiers, exclusion criteria, annotation materials, and analysis scripts, which is the right norm to endorse.
There is also a deeper evidential question the paper handles carefully but which deserves emphasis. Model rationales are treated as expressed moral grounds, not as evidence of internal reasoning. Interpretability research has shown repeatedly that plausible post-hoc explanations need not be faithful to the mechanisms driving model outputs. The paper's framing is defensible: what matters for transparency and contestability is what moral grounds a model makes available for human inspection, regardless of whether those grounds causally produced the output. But this means rationale divergence is a claim about the model's expressed moral vocabulary, not about its underlying computations.
Several questions remain open. Do rationale distributions shift with prompting strategy? The paper covers one protocol, and there is good reason to expect that chain-of-thought prompting, role assignment, or explicit moral-framework instructions would change the picture. Whether such shifts produce genuine realignment or surface-level vocabulary change is itself an empirical question worth pursuing. The GaMS results also invite follow-up: since all items are in English, the open-model comparisons here are family comparisons rather than multilingual ones, and testing these models on Slovene moral scenarios would be a natural extension.
Implications for Alignment Evaluation
The paper's methodological argument is the most transferable contribution. Agreement metrics are not wrong, they are incomplete. They measure output convergence efficiently and at scale. What they cannot detect is whether that convergence is supported by moral grounds that are human-recognisable, inspectable, and contestable in the way that good ethical reasoning should be.
The practical implication is that evaluation pipelines for ethically sensitive applications should include rationale-aware diagnostics alongside label agreement. This does not require models to reproduce every human rationale, nor does it treat human majority labels as moral ground truth. It asks a more modest but more informative question: does the model's expressed moral vocabulary overlap enough with human moral categories that disagreements can be identified, articulated, and challenged?
For the field, this paper makes a contribution that is easy to understate. It does not claim that current models are dangerously misaligned or that RLHF is fundamentally broken. It claims, with empirical support, that the standard evaluation proxy is systematically incomplete. That is a quieter finding, but it has direct consequences for how alignment claims should be qualified and how evaluation datasets should be designed going forward.
The full paper is available at arxiv.org/abs/2608.12368.