Knowledge distillation from closed-source models like GPT-4 presents a fundamental technical obstacle: the KL-divergence term in the standard distillation objective requires access to the teacher's output distribution, which API-gated models simply do not expose. Most existing work sidesteps this by treating the teacher's sampled outputs as hard labels and fine-tuning a student model directly on those samples. This is instruction fine-tuning under a different name, and it discards a substantial portion of the information the teacher encodes at inference time. A recent paper from arXiv proposes a more principled alternative: estimate the teacher's token-level output distributions using a Bayesian framework, then run standard soft-label distillation against those estimates.
The Core Problem with Hard-Label Distillation
To appreciate what is being attempted here, it helps to be precise about what gets lost when you discard soft labels. In the standard Hinton-style distillation objective, the loss decomposes into a cross-entropy term against the teacher's argmax prediction and a KL-divergence term against the teacher's full distribution over the vocabulary. The KL term is where the real signal lives. A teacher assigning 0.6 probability to token A and 0.3 to token B is conveying something qualitatively different from one that assigns 0.99 to A, even when both produce A as their sampled output. That distributional information encodes uncertainty, near-synonyms, alternative phrasings, and structured uncertainty about the task, none of which survives the sampling step.
When you fine-tune on GPT-4 outputs, you recover only the argmax behaviour. The student learns to reproduce surface-level response patterns but has no signal about the confidence structure of the teacher's predictions. This is not a minor limitation. For reasoning tasks in particular, the difference between a model that is confidently right and one that is uncertainly right is diagnostically important and propagates through multi-step inference.
Bayesian Estimation as a Workaround
The proposed method frames the problem as Bayesian estimation of the teacher's latent output distribution. The two-stage procedure works as follows:
- Prior estimation: The corpus of text generated by the closed-source teacher is used to construct an initial approximation of the teacher's token-level distribution. The intuition is that the generated corpus encodes coarse-grained information about which tokens the teacher finds plausible in context, even if individual samples are draws from that distribution rather than the distribution itself.
- Posterior estimation: A proxy model, in the experiments LLaMA-33B fine-tuned on the same corpus, is used to refine the prior. The proxy is first aligned to the teacher by instruction fine-tuning on the generated corpus, then used to produce soft labels that update the prior distribution. The resulting posterior serves as the estimated teacher distribution for the KL term in the distillation objective.
The student, LLaMA-7B or LLaMA-13B, is then trained against this estimated distribution using the standard objective. The proxy model acts as a differentiable bridge between the black-box teacher and the open student, providing a tractable approximation to the soft labels that would otherwise be inaccessible.
This is a sensible construction. If the proxy is well-aligned with the teacher, its output distribution should be a reasonable surrogate for the teacher's. The Bayesian framing is somewhat loose in the paper, the prior and posterior are not derived from a fully specified generative model, but the procedural logic is sound: start with a corpus-derived estimate, refine it with a capable aligned model.
Experimental Results and What They Actually Show
The empirical evaluation spans six benchmarks: BBH, ARC, AGIEval, MMLU, CSQA, and GSM8K. Using GPT-4 as the teacher, LLaMA-33B as the proxy, and LLaMA-7B/13B as students, the method achieves average improvements from 36.31 to 39.43 (7B) and 43.37 to 46.08 (13B) over direct instruction fine-tuning baselines. The gains are consistent across benchmarks, though not uniform. Mathematical reasoning on GSM8K shows some of the cleaner improvements, which makes sense given that the distributional information about intermediate reasoning steps is particularly valuable there.
The ablation study is informative. Prior estimation alone contributes modest gains. Posterior estimation contributes more substantially. The combined system outperforms either component alone, which at minimum confirms that the two estimation stages are not redundant. The proxy model selection experiment is also worth noting: a LLaMA-33B proxy outperforms a LLaMA-13B proxy as the bridge, which is expected but good to verify empirically.
There are some results that warrant closer scrutiny. The FlanT5 experiments show much smaller improvements, which the authors attribute to FlanT5 already being instruction-tuned. This is plausible, but it also raises the question of whether the gains in the LLaMA experiments are partly attributable to the baseline being a relatively weak starting point. LLaMA-7B without instruction tuning is not a strong baseline, and the gap between hard-label fine-tuning and the proposed method might narrow considerably if the baseline were a more capable instruction-tuned model.
The continuous fine-tuning experiment on the proxy model reveals an important practical constraint: over-training the proxy on the corpus degrades its effectiveness as a soft-label provider, likely due to overfitting that reduces the diversity of its output distributions. This places a real operational constraint on how the proxy must be trained, and it is not entirely clear from the paper how sensitive the final results are to this hyperparameter.
Limitations and Open Questions
Several issues deserve attention before treating this as a general solution to black-box distillation.
First, the quality of the estimated soft labels is bounded by the quality of the proxy model. LLaMA-33B is a capable model, but it sits at roughly 45.7 average performance against GPT-4's 75.5 on the reported benchmarks. The proxy's output distribution may be a poor approximation to the teacher's in precisely the regions where the teacher is most capable, which are the regions where accurate soft labels would be most valuable. The method is likely most effective when the proxy is close to the teacher in capability, which is a non-trivial requirement.
Second, the Bayesian framing, while conceptually useful, is not fully formalised. The prior is derived from corpus statistics in a way that is not explicitly specified as a probability model over distributions. The posterior update via the proxy is similarly informal. A more rigorous treatment would specify the prior family, the likelihood model, and the update rule explicitly. Without this, it is difficult to reason about when the estimation will be accurate and when it will fail.
Third, the method inherits the data dependency of all instruction fine-tuning approaches. The corpus used for proxy alignment, prior estimation, and student distillation is identical across all stages, which raises questions about information leakage and whether the improvements are partly attributable to the proxy simply memorising the training distribution rather than genuinely capturing the teacher's generalisation behaviour.
Fourth, the computational cost is non-trivial. Running inference through LLaMA-33B to generate soft labels for every training example adds substantial overhead compared to simple fine-tuning. The paper does not provide a detailed cost comparison, which makes it difficult to assess the practical trade-off between performance gain and compute expenditure.
Broader Implications for Open-Source Model Development
The broader research context here is significant. The gap between closed-source frontier models and open-source alternatives is a persistent concern for reproducibility, auditability, and equitable access to capable AI systems. Methods that can close this gap without requiring white-box access to the teacher are genuinely valuable, provided they work reliably.
This paper sits within a growing body of work on black-box distillation and model alignment transfer, alongside approaches like RLHF from AI feedback, constitutional AI, and various forms of self-improvement. The distinguishing feature of this approach is its explicit attempt to recover distributional information rather than just behavioural imitation. That is the right problem to be working on, even if the current solution is incomplete.
The practical ceiling for this approach is probably set by the proxy model's capability relative to the teacher. As open-source models continue to close the gap with closed-source ones, the proxy approximation will improve, and methods like this will become more effective. There is a reasonable argument that the value of this approach increases over time as the proxy quality improves, which gives it a degree of future-proofing that purely behavioural imitation methods lack.
For practitioners working on small-model deployment where API costs are a constraint, the framework offers a structured way to think about the distillation pipeline. The key engineering decision is proxy model selection, and the empirical evidence suggests that investing in a larger, better-aligned proxy pays dividends in student performance. Whether those dividends justify the additional compute relative to simpler fine-tuning baselines will depend heavily on the specific deployment context and the benchmarks that matter for the application.