One of the quieter but consequential problems in modern LLM development is deciding which data to train on during supervised fine-tuning. With instruction datasets now routinely reaching millions of samples, training on everything is both computationally wasteful and, counterintuitively, sometimes harmful to downstream performance. The standard response is data selection: pick a small, high-quality subset and fine-tune on that instead. The question is how to define quality, and for whom.
A new paper, Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training, argues that existing approaches get this wrong in a fundamental way. They treat sample quality as a static property, independent of the model being trained. Data-DPO proposes instead to ask which samples are most compatible with the current capability distribution of the specific target model, and builds a lightweight selection pipeline around that question.
The Core Problem with Static Data Valuation
Most existing data selection methods fall into two camps: importance-based methods (e.g. LESS, EL2N, TIVE) that estimate each sample's contribution to model updates or downstream performance, and diversity-based methods (e.g. PRISM, SemDeDup) that reduce redundancy through embedding-space coverage. Several recent methods combine both signals into a unified scorer.
The shared assumption across all of these is that a sample's value can be determined without reference to the model being fine-tuned. The authors challenge this directly. SFT does not teach models new knowledge from scratch; it activates, calibrates, and aligns capabilities that already exist from pretraining. A sample that is trivially easy for a strong model may be exactly the right difficulty for a weaker one, and vice versa. The same reasoning applies across model families: architectural differences mean that two models with similar benchmark scores may have quite different internal capability distributions, and therefore different training needs.
This framing is not entirely novel, but the proposed solution is more practical and principled than most prior attempts to address it.
Methodology: Preference Learning from One-Step Probing
Data-DPO proceeds in three stages.
First, a small representative probe set is sampled from the full candidate data. The target model performs a single gradient update on each probe sample, and the change in activation patterns before and after this update is recorded. Samples that produce larger, more consistent activation changes are treated as more compatible with the model's current learning state. These activation differences are then converted into pairwise preference relations between samples, giving a dataset of the form: the target model prefers sample A over sample B.
Second, a lightweight reward model (a residual MLP in the default configuration) is trained on these pairwise preferences using a DPO-style objective. This reward model learns to score arbitrary candidate samples according to how well they align with the target model's inferred preferences, without requiring additional probing on the full dataset.
Third, the final subset is constructed by combining three signals: the learned target-model preference score, an external quality score from a small VLM (Qwen3-VL-4B or Qwen3.5-9B depending on dataset), and a marginal diversity term that penalises redundancy. These are combined into a single selection objective optimised greedily under the budget constraint.
The one-step probing design is worth emphasising. Many prior methods use a proxy model, trained on a small random subset, as a stand-in for the original target model. The authors test this directly and find it degrades performance substantially, especially at small budgets. Even a small amount of SFT shifts the model's capability distribution enough that its data preferences diverge from those of the original pretrained model. This is a clean and practically important finding.
Results and What They Show
Experiments are conducted on two multimodal datasets: Vision-Flan (general instruction tuning) and LLaVA-CoT (reasoning-oriented fine-tuning), with target models LLaVA-V1.5-7B and Llama-3.2-11B-Vision-Instruct respectively. Performance is reported as Average Relative Performance (ARP) against full-data training across 12 benchmarks per dataset, under 5%, 10%, and 15% data budgets.
Key results:
- On Vision-Flan, Data-DPO achieves ARP of 100.76, 102.63, and 102.70 at 5%, 10%, and 15% budgets, all exceeding full-data training and outperforming the next-best baseline by 1.31 to 2.35 ARP points.
- On LLaVA-CoT, Data-DPO achieves 102.73 and 103.93 ARP at 5% and 10% budgets, with improvements of 3.60 and 4.14 over the strongest competing method.
- At the 15% budget on LLaVA-CoT, EL2N slightly outperforms Data-DPO (102.46 vs 101.31), though Data-DPO still exceeds full-data training and outperforms all other baselines.
- Robustness checks with LLaVA-v1.5-13B and Qwen2VL-2B-Instruct confirm consistent gains over random selection across model scales and families.
- Ablations show that all three selection signals (preference, quality, diversity) contribute positively, and that a simpler plain MLP reward model still outperforms random selection, confirming that the gains are driven by the preference supervision rather than model capacity.
The consistent ability to exceed full-data training performance at small budgets is notable. It suggests that the full datasets contain a non-trivial proportion of samples that are actively unhelpful for specific target models, and that model-aware selection can identify and exclude them.
Limitations and Open Questions
The authors are candid about several limitations. The method inherits biases from its external quality scorer and embedding model, and performance could degrade if these signals are poorly calibrated for a given domain. The one-step probing cost, while much cheaper than training a proxy model, still adds overhead that scales with probe set size. The method is also evaluated exclusively on multimodal models, and it is not immediately clear how the activation-difference signal would behave in text-only settings where the capability distribution may be more uniform.
There are also questions the paper does not fully address. The probe set is sampled to be representative, but the sensitivity to probe set composition is not analysed in depth. It is also worth asking how the method performs when the candidate data distribution is deliberately diverse or adversarially constructed, rather than drawn from a coherent instruction-following corpus. The authors acknowledge that distribution shift scenarios may limit effectiveness.
More broadly, the framing raises an interesting question for the field: if data value is model-dependent, does this mean selection pipelines need to be re-run for every new model checkpoint, or can preference signals transfer across closely related models? The robustness experiments suggest some degree of transferability across model families, but this is not systematically explored.
Data-DPO is a well-motivated and practically grounded contribution to the data selection literature. The core insight, that SFT data value is relational rather than intrinsic, is correct and underexplored, and the proposed mechanism for operationalising it is both efficient and effective. The results across two datasets and multiple budgets are convincing. For anyone working on instruction tuning pipelines, particularly in multimodal settings, this is worth reading carefully.
Full paper: arxiv.org/abs/2608.16926