Privacy and fairness are two of the most pressing concerns in modern NLP, and they are rarely studied together with the rigour each deserves. A new paper from Eduardo Tenorio, Karuna Bhaila, and Xintao Wu, available on arXiv, takes a systematic look at what happens to social bias when you train a large language model with differential privacy. The findings are nuanced and, in some respects, cautionary: privacy guarantees do not straightforwardly translate into fairer models.
The practical motivation is clear. LLMs trained on web-scale corpora are known to memorise fragments of their training data, creating real risks of sensitive information leakage. Differential privacy, specifically the DP-SGD algorithm, has become the standard principled response to this problem. By clipping per-sample gradients and adding calibrated Gaussian noise during training, DP-SGD limits how much any individual data point can influence the final model. What has received far less attention is whether this mechanism, which effectively down-weights or obscures the influence of outlier data points, also affects the social stereotypes and biases that those data points might encode.
What the Paper Contributes
The core contribution is methodological breadth. Rather than evaluating bias through a single benchmark, the authors assess a pretrained LLM fine-tuned with DP-SGD across four distinct evaluation paradigms:
- Sentence scoring: comparing model-assigned likelihoods to stereotypical versus anti-stereotypical sentence pairs, following the approach used in benchmarks like StereoSet and CrowS-Pairs.
- Text completion: prompting the model to generate continuations and analysing the outputs for demographic associations and stereotypical content.
- Tabular classification: using the LLM as a classifier on structured datasets where protected attributes are present, examining whether DP affects disparate treatment across groups.
- Question answering: probing the model's responses to questions that implicitly invoke social categories, measuring consistency and bias in generated answers.
This multi-paradigm design is the paper's strongest methodological choice. Bias in LLMs is not a single phenomenon; it manifests differently depending on whether you are looking at token probabilities, generated text, or downstream classification behaviour. A paper that collapses these into a single metric risks missing the full picture.
Key Findings
The headline result is that DP training does reduce bias in sentence scoring tasks. When bias is operationalised as the difference in log-likelihoods between stereotypical and neutral or counter-stereotypical sentence pairs, the DP model shows measurable improvement over the non-DP baseline. This is a plausible finding: DP-SGD's gradient clipping and noise injection may reduce the model's sensitivity to the specific distributional patterns that produce skewed likelihood ratios.
The more significant finding, however, is that this improvement does not generalise. Across text completion, tabular classification, and question answering, the DP model does not consistently exhibit less bias than its non-DP counterpart. The authors characterise this as a discrepancy between logit-level bias and output-level bias, and it is a distinction worth taking seriously.
Logit-level bias, as measured by sentence scoring, reflects something about the model's internal probability distributions over token sequences. Output-level bias, as measured by generation and classification tasks, reflects the actual behaviour a user or downstream system would encounter. The two can diverge for several reasons: decoding strategies, temperature, and instruction formatting all mediate the relationship between raw logits and final outputs. A model that looks less biased at the probability level may still produce stereotypical completions because the bias signal, while attenuated, is not eliminated and can be amplified by sampling dynamics.
Perhaps the most important finding for the broader privacy-fairness debate is that reducing memorisation does not necessarily reduce unfairness. This challenges an implicit assumption that sometimes appears in the literature, namely that if DP prevents a model from memorising specific sensitive data points, it should also prevent the model from learning the biased associations those data points encode. The paper's results suggest the relationship is considerably more complex. Bias in LLMs is a statistical property of the training distribution as a whole, not merely a consequence of memorising individual examples. DP-SGD perturbs individual gradient contributions but leaves the aggregate distributional signal largely intact, particularly when training data is large and biases are pervasive.
Limitations and Open Questions
The paper is honest about its scope. The evaluation is conducted on a single pretrained model architecture fine-tuned with DP-SGD, which limits the generalisability of the findings across model families, scales, and training regimes. It is not clear whether the same discrepancy between logit-level and output-level bias would appear in much larger models, where emergent capabilities can change the relationship between internal representations and surface behaviour in unpredictable ways.
The choice of epsilon, the privacy budget parameter in DP-SGD, also matters considerably. Tighter privacy guarantees require more noise, which could in principle have stronger effects on bias, though likely at the cost of greater utility degradation. The paper does not provide a detailed sweep across epsilon values, which would help clarify whether the null results on output-level bias are robust across different privacy-utility trade-off points or specific to a particular regime.
There is also a question about what the sentence scoring results actually measure. Benchmarks like StereoSet and CrowS-Pairs have themselves been criticised for encoding a particular, often Western and English-centric, conception of what counts as a stereotype. If DP reduces bias on these benchmarks, it is worth asking whether that reduction reflects genuine fairness improvement or an artefact of how those benchmarks operationalise the concept.
Implications for the Field
The practical implication for practitioners is straightforward: do not assume that applying DP to your LLM training pipeline will incidentally improve fairness. The two objectives require separate attention and separate mitigation strategies. If anything, the paper suggests that DP could create a false sense of security if fairness is evaluated only through sentence scoring, which is the most common approach in the literature.
For researchers, the paper makes a strong case for multi-paradigm evaluation as a methodological norm rather than an optional addition. Given the computational cost of training DP models, there is always pressure to evaluate on a small number of benchmarks. This paper shows that such shortcuts can produce misleading conclusions about the fairness implications of privacy interventions.
The broader tension the paper surfaces, between privacy, fairness, and utility as potentially competing objectives in LLM training, remains an open research problem. Understanding the theoretical mechanisms by which DP-SGD affects learned representations, and whether those effects can be steered to simultaneously improve both privacy and fairness, is a question that deserves considerably more attention than it has received so far.
The full paper is available at arXiv:2605.11195.