Most stereotype benchmarks used to evaluate large language models were built with English-speaking, predominantly American, cultural contexts in mind. Datasets like StereoSet and CrowS-Pairs remain the field's primary reference points, yet they capture a narrow slice of global social bias. This paper, available at arXiv:2607.07895, addresses that gap directly by proposing a scalable annotation framework and applying it to produce EspanStereo, a stereotype examination dataset covering five Spanish-speaking countries: Spain, Mexico, Argentina, Colombia, and Nicaragua.
The core problem is not just translation. Prior work that converted English stereotype datasets into other languages retained the cultural assumptions baked into the originals. A dataset translated from American English into Spanish will still encode assumptions about US racial dynamics, US religious demographics, and US political culture. EspanStereo is constructed natively, which turns out to matter quite a lot.
Key Contributions
The paper makes two separable contributions that are worth distinguishing. The first is methodological: a human-LLM collaborative framework for stereotype acquisition that substantially reduces annotation cost. The second is a concrete dataset produced by applying that framework.
On the methodological side, the key insight is that stereotype elicitation, the most expensive phase of dataset construction, can be offloaded to an LLM. Because stereotypes are, by definition, widely circulated beliefs, they appear frequently in training corpora. GPT-4o is used to generate candidate stereotype lists for each country, which in-culture human annotators then validate by majority vote. Annotators are not asked to generate stereotypes from scratch, only to confirm or reject generated candidates. This is a meaningful reduction in cognitive load and recruitment difficulty, particularly for smaller populations like Nicaragua where sourcing large annotator pools is genuinely hard.
The resulting dataset contains 538 validated stereotypes across five countries, covering race, religion, gender, sexual orientation, and age categories. All stereotypes were originally written in Spanish.
Methodology in Detail
The pipeline has three stages. First, GPT-4o generates a list of candidate stereotypes for a given country and category. Second, five in-culture annotators per country validate each candidate, with majority voting determining inclusion. Third, annotators instantiate validated stereotypes into sentence-level examples suitable for probing transformer models.
Validation rates are generally high: Mexico and Spain both achieve 97% overall validation, Argentina 92%, Colombia 86%. Nicaragua is the outlier at 71%, driven by poor performance in the race category where the model generated immigration-related stereotypes that annotators did not recognise as locally common. This is an informative failure mode. LLMs appear to generalise across Latin American countries in ways that flatten real distinctions, particularly around immigration narratives.
The paper uses the probing-and-pruning approach from Ma et al. (2023) to evaluate stereotype encoding in XLM-R and BETO, two transformer models with Spanish support. This choice is appropriate given the research goal, though it does mean the evaluation is constrained to encoder-only architectures rather than the generative models now dominant in deployment.
Results and What They Show
The cross-country overlap analysis is one of the more striking sections. Pairwise stereotype overlap between countries rarely exceeds 21%, and in race and religion categories it frequently drops to single digits. A few examples illustrate why this matters:
- Spain is the only country with stereotypes targeting Romani people and Moroccans, reflecting contemporary tensions specific to Spanish society.
- Nicaragua is the only country where stereotypes about the Creole population appear, an ethnic group concentrated on the Caribbean coast with a distinct history from the Spanish-speaking majority.
- Colombia, Argentina, and Mexico all contain stereotypes about indigenous land conflicts, but the specific groups and historical contexts differ entirely: the Wayuu and Cerrejón mine in Colombia, the Mapuche and the Conquest of the Desert in Argentina, the Mixtec and the Leyes de Reforma in Mexico.
These are not superficial differences. They reflect genuinely distinct historical and political trajectories that a translation-based approach would erase. The dataset's ability to capture this granularity is the strongest argument for the methodology.
The LLM evaluation results show significant variation in stereotypical behaviour across the five countries, which the authors take as evidence that country-specific assessment is necessary. XLM-R and BETO encode stereotypes at different rates depending on the country, and these patterns do not correlate in obvious ways with country population or data availability. This is a useful finding for practitioners deploying Spanish-language models across Latin America.
Limitations and Open Questions
The authors are candid about coverage limitations. LLMs are better at capturing established, well-circulated stereotypes than emerging or subcultural ones. Social media analysis or domain expert consultation could fill this gap, but neither is integrated into the current framework. This is a reasonable scope limitation for a first dataset, but it means EspanStereo likely skews toward stereotypes that are already legible to mainstream discourse.
There is also a question about the LLM used for generation. GPT-4o's training data distribution is not public, and its coverage of, say, Nicaraguan cultural context is almost certainly thinner than its coverage of Mexican or Spanish context. The lower validation rate for Nicaragua is consistent with this. Future work might explore whether open-weight models with different training mixes produce meaningfully different stereotype lists.
The evaluation is restricted to encoder models. Given that most deployed Spanish-language applications now use instruction-tuned generative models, extending the benchmark to evaluate models like Mistral, LLaMA, or GPT-4o in generative stereotype tasks would substantially increase practical relevance.
Finally, the inter-annotator disagreement question is handled somewhat briefly. The paper cites Uma et al. (2021) to argue that disagreement in subjective tasks reflects genuine cultural variation rather than noise, which is a defensible position. But the dataset reports that nearly all stereotypes were validated by 4/5 or 5/5 annotators, which actually suggests relatively high agreement. A more detailed breakdown of disagreement patterns, particularly in categories like sexual orientation where cultural attitudes vary sharply, would strengthen the methodological claims.
Despite these open questions, EspanStereo fills a real gap. The framework is language-agnostic and the paper makes a credible case that it could be applied to other underrepresented language communities. For researchers working on multilingual fairness evaluation or culturally grounded NLP, this is a useful methodological reference point.
Full paper: arXiv:2607.07895