There is a growing body of evidence that instruction-tuned language models infer demographic attributes from subtle linguistic cues and condition their outputs on those inferences, even when the attributes are never explicitly stated and even when the model would refuse to answer a direct question about them. This creates a troubling gap: alignment training suppresses verbalization of demographic inference without necessarily suppressing the inference itself. A new pilot study on arXiv (2607.21774) takes a careful look at one specific instance of this phenomenon, asking whether Qwen2.5-7B-Instruct internally represents Colombian identity when processing prompts containing only implicit cues, using a training-free verbalizer to inspect the model's residual stream before any output is produced.
The motivation is well-grounded. Colombian Spanish sits near the bottom of dialect recognition benchmarks (F1 = 0.282, compared to 0.723 for Peninsular Spanish), a disparity that tracks training data composition rather than the actual digital availability of the variety. If a model is already forming a nationality inference from indirect cues, that latent representation could shape generation quality, register, and content in ways that compound existing disparities for underrepresented varieties. Yet no prior work had inspected what the model represents internally, before output, when processing Colombian Spanish prompts.
Methodology
The authors adopt Natural Language Autoencoders (NLA), a technique introduced by Fraser-Taliente et al. (2026) in which a trained verbalizer translates residual-stream activations into free-text descriptions of what the model appears to be representing at a given position. The key appeal here is that NLA requires no supervised probing classifier and no labelled demographic dataset; it reads off a natural-language description directly from the activation vector. The authors probe layer 20 of Qwen2.5-7B-Instruct, sampling activations at four positional quartiles within each prompt.
The dataset is small but deliberately structured: 30 prompts arranged as 15 matched Spanish-English pairs, divided into three groups of 10:
- Explicit cues: prompts that name Colombia or Colombian Spanish directly
- Implicit cues: prompts containing a single indirect lexical marker of Colombian identity without naming the country
- Neutral controls: prompts with no Colombian cues
For each (group × quartile) cell, they compute the rate at which NLA explanations mention nationality, report Wilson score intervals appropriate for the small per-cell counts (n = 9-10), and test implicit-versus-neutral differences at each quartile with Fisher's exact test. The quartile of first appearance is compared across groups with a Mann-Whitney U test. The authors are explicit that this is an exploratory design: all tests are two-sided and uncorrected for multiple comparisons, and the study is framed as generating directional evidence rather than confirming effects.
Key Findings
The central result is that the implicit-cue group shows a monotonically rising nationality mention rate across quartiles: 0.10, 0.20, 0.50, 0.78. The neutral control remains flat and low. The explicit group reaches ceiling quickly (0.10 at Q1, then 0.00 at Q2 before jumping to 1.00 at Q3 and Q4), which the authors trace to a known NLA failure mode where the verbalizer confabulates a contextually wrong country name at early positions. Once they restrict to Colombia-specific citations rather than any nationality mention, the implicit-versus-neutral separation reaches conventional significance at the final quartile (p = 0.023), with the implicit group at 0.20 and the neutral control at 0.00 throughout Q1-Q3.
Two apparent irregularities in the raw data, the explicit group's Q2 drop and the neutral group's unexpected Q4 spike to 0.20, are both resolved by the same correction. This is a useful methodological contribution in its own right: NLA explanations used as regional-identity probes need to be filtered for country specificity, not just any nationality mention, to avoid confabulation artefacts inflating apparent detection rates.
Limitations and What Remains Open
The authors are admirably candid about the study's constraints. With n = 5 scenarios per cell after exclusions, the p = 0.023 result at Q4 sits at the edge of what any reasonable reader should treat as directional rather than confirmatory. The study probes a single layer (layer 20) and a single model; whether this depth is particularly diagnostic for nationality inference, or whether the effect generalises across architectures, is entirely unknown. The 30-prompt dataset is too small to support subgroup analysis by prompt type, language (Spanish vs. English within the implicit group), or specific lexical cue.
There is also a gap between what NLA explanations measure and what we might ultimately want to know. The verbalizer surfaces what the residual stream appears to represent in natural language terms, but it does not tell us whether that representation causally influences downstream generation. The alignment gap literature (Bouchaud and Ramaciotti, 2025; Neplenbroek et al., 2025) provides strong behavioural evidence that latent demographic inferences do shape outputs, but connecting the specific NLA-detected nationality representation to a measurable generation disparity would require a separate causal intervention experiment, something this pilot does not attempt.
The implicit cue used across the 15 pairs is also not described in detail in the available abstract and results sections, which makes replication harder. If the cue is a single lexical item (a Colombian-specific colloquialism, for instance), the generalisability to broader implicit Colombian-identity signals is unclear.
Why This Work Matters
Despite its scale, this paper does something that larger bias-audit studies have not done: it connects activation-level interpretability directly to a specific underrepresented Spanish variety, using an unsupervised verbalizer rather than a supervised probe trained on demographic labels. That design choice matters because supervised probes can, in principle, detect linear demographic structure that is not functionally relevant to generation; NLA explanations, while noisier, are at least grounded in the same representational space the model uses to produce output.
The study also makes a concrete methodological recommendation, filtering NLA outputs for country-specific citations when using the technique as a regional-identity probe, that should be directly useful for anyone extending this line of work. The suggested next steps are sensible: higher n per cell, a second residual-stream layer for comparison, and ideally a causal follow-up linking the detected representation to output quality differences for Colombian Spanish prompts.
For a field where most interpretability work focuses on English and most bias work focuses on behaviour rather than internal representations, this is a useful proof of concept. The finding that a single implicit cue can drive a rising nationality-mention rate across prompt quartiles, even if not yet conclusively established at this sample size, is worth taking seriously as a hypothesis to test at scale.
The full paper is available at https://arxiv.org/abs/2607.21774.