One of the more persistent frustrations with in-context learning (ICL) is how sensitive it is to the choice of demonstrations. Pick the wrong examples and performance can fall below random selection. This sensitivity is manageable when you have access to target-domain data, but in many real deployments you simply do not. A model trained on product reviews gets queried on medical text; a sentiment classifier built on news is applied to social media. The target distribution is unknown, and the standard retrieval playbook breaks down.
This paper, available on arXiv, proposes DOPA (Demonstration Optimisation via OOD Proxy Assessment), a retrieval framework designed specifically for this target-inaccessible setting. The core idea is to construct a proxy for the unknown target distribution using only the source-domain model and its untuned counterpart, then use that proxy to score and select demonstrations.
The Problem Worth Taking Seriously
The ICL literature has produced a range of retrieval strategies: BM25-based term matching, sentence embedding similarity (SBERT, SimCSE, RoBERTa), influence-function scoring, and diversity-aware selection. Most of these assume either that the retrieval pool is distributionally close to the test instance, or that a fine-tuned retriever can be trained with LLM feedback. Neither assumption holds when the target domain is off-limits.
Data augmentation approaches, which rewrite source samples to resemble target instances, face the same fundamental barrier. Without target examples to guide the rewriting, the adapted samples can drift in unhelpful directions. The BOSS benchmark used in this paper was designed precisely to stress-test this scenario, covering sentiment analysis, toxicity detection, natural language inference, and named entity recognition across genuinely shifted distributions.
Methodology
DOPA builds its target-domain approximation from two models. The source proxy is an instruction-tuned version of a base LLM, fine-tuned on source-domain data via LoRA. The target proxy is the same base model left unmodified, which retains broad prior knowledge not specific to the source domain. For any source-domain sample, the ratio of perplexities between these two models provides an OOD score: samples that the source proxy finds much more natural than the base model are strongly source-domain-specific; samples where the ratio is closer to one are more likely to generalise.
This perplexity ratio approach draws on likelihood-ratio methods for OOD detection, and the authors provide a bounded proxy error analysis to justify its theoretical soundness. The bound establishes that the approximation error between the proxy-estimated OOD score and the true score is controlled, which is a necessary sanity check even if the bound itself is not tight in practice.
Demonstration selection then combines three signals:
- OOD proxy score: how well a source sample generalises beyond the source distribution
- Representational similarity: standard embedding-based proximity to the test instance
- Mahalanobis distance diversity constraint: a global constraint that discourages selecting demonstrations that cluster together in representation space
The diversity component is worth attention. Many retrieval methods optimise per-sample relevance independently and end up with redundant demonstration sets. The Mahalanobis distance formulation accounts for the covariance structure of the representation space rather than treating dimensions as independent, which is a more principled approach than simple cosine-distance deduplication.
For closed-source models like GPT-4o-mini and GPT-3.5-turbo, where internal representations are unavailable, the authors substitute LLaMA 3.2-3B as the proxy pair. This is a pragmatic workaround, though it introduces an obvious mismatch between the proxy and the model being prompted.
Results
Experiments span six open-weight models (GPT2-xl, Qwen3-1.7B, Qwen3-8B, Gemma2-2B, LLaMA3.2-3B, LLaMA3.1-8B) and two closed-source APIs, evaluated on the BOSS benchmark. Several findings stand out:
- DOPA outperforms all baselines on classification tasks, with Wilcoxon signed-rank tests confirming statistical significance at p ≤ 0.05 across nine evaluation tasks for the closed-source models.
- On NER, gains are more pronounced. DOPA achieves 44.84% average accuracy on GPT-4o-mini versus 34.09% for random selection and 34.08% for KNN. The gap is similarly large for GPT-3.5-turbo.
- Random selection consistently beats KNN under LLaMA3.2-3B for classification, which is a striking result that underlines how badly semantic similarity can misfire under distribution shift.
- The Rewrite baseline performs poorly throughout, confirming that augmentation without target-domain examples is unreliable.
- InfICL is competitive in isolated settings but highly unstable, collapsing on NLI with GPT2-xl and LLaMA3.2-3B.
The ablation study (Table 4) isolates each component's contribution. Removing the Mahalanobis diversity constraint (DOPA-mah) degrades performance modestly. Removing the similarity signal (DOPA-sim) causes larger drops, suggesting similarity remains informative even under shift. Removing the proxy entirely (DOPA-pro) produces consistent but smaller degradation than removing similarity, which is somewhat surprising and worth investigating further. The variant using uniform sampling over proxy-selected candidates (DOPA-uni) also underperforms the full model, confirming that joint optimisation matters.
Limitations and Open Questions
The authors are candid about the central limitation: the target proxy is an approximation, and if the source and target distributions are very far apart, the untuned base model may not adequately represent the target. This is not a flaw unique to DOPA, but it does mean the method's reliability degrades in proportion to how exotic the target domain is relative to the base model's pretraining data.
The closed-source proxy substitution raises a related concern. Using LLaMA 3.2-3B as a stand-in for GPT-4o-mini introduces a mismatch that the authors acknowledge but do not fully characterise. It would be useful to know how sensitive results are to the choice of proxy model, particularly whether smaller or weaker proxies systematically bias the OOD scores in a particular direction.
The evaluation is also restricted to the BOSS benchmark. BOSS is well-designed for OOD stress-testing, but it covers a specific slice of NLP tasks. Extension to generation tasks beyond NER, to multilingual settings, or to domains with more extreme covariate shift (e.g., code or clinical text) would clarify the method's scope.
There is also a computational cost question that the paper does not address directly. Instruction-tuning a proxy model per source domain adds overhead. For settings where source domains change frequently, or where multiple source domains exist, the cost of maintaining proxy pairs could become significant.
Despite these caveats, DOPA addresses a real and underserved problem. Most demonstration retrieval research assumes some form of target-domain access, whether explicit test instances or distribution statistics. The target-inaccessible setting is arguably the more common deployment scenario, and the perplexity ratio proxy is a clean, theoretically motivated solution. The code is available at github.com/bort64/ood_code.