Zero-shot image classification with vision-language models like CLIP depends critically on how class labels are embedded into text prompts. The standard approach of averaging embeddings across multiple prompt templates (prompt ensembling) is known to be sensitive to prompt quality, and the question of how to weight different prompts optimally has attracted considerable attention. This paper identifies a structural limitation in existing weighting strategies: they assign the same weight vector to every class, implicitly assuming that prompt relevance is class-independent. CARPRT (Class-Aware Zero-Shot Prompt Reweighting) addresses this directly by estimating a separate weight vector per class, using only unlabelled inference data and black-box forward passes through the VLM.
The motivation is straightforward once stated. A prompt like "an aerial view of" is plausibly useful for classifying airports but actively misleading for classifying apples. Existing methods, including the Weighted Prompt Ensembling (WPE) approach of Allingham et al. (2023), estimate a single global weight vector that must serve all classes simultaneously. The paper provides both empirical and theoretical justification for why this is suboptimal: a controlled experiment on Flowers102 shows that applying WPE independently per class (using ground-truth labels as an oracle) consistently improves accuracy, and the optimal weights vary substantially across classes.
Theoretical Framing
The authors cast prompt ensembling within a probabilistic framework using Bayes' theorem, modelling the posterior over prompt weight matrices given the unlabelled dataset and the prompt set. Class-conditional likelihoods are handled via energy-based models, interpreting cosine similarity scores as negative energies. Within this framework, they show formally that class-agnostic weighting schemes implicitly impose a conditional independence assumption between class labels and prompt weights given an image. This assumption is generally violated in practice, which provides a principled account of why class-aware reweighting should help.
This theoretical grounding is one of the paper's stronger contributions. Rather than simply ablating an engineering choice, the authors situate CARPRT within a coherent probabilistic account of what prompt ensembling is actually doing, which makes the method's design choices easier to interpret and extend.
Methodology
CARPRT operates in two steps at inference time, requiring no gradient computation or access to model internals beyond similarity scores.
- Score Calculation: For every image in the unlabelled reference set, similarity scores are computed against all prompt-class combinations. Each image is assigned a pseudo-label corresponding to the highest-scoring combination.
- Weight Calculation: For each class, the weight assigned to a given prompt is derived from the maximum similarity that prompt achieves with images pseudo-labelled to that class. These raw scores are then normalised (with a temperature parameter) to produce a proper weight distribution over prompts for each class.
The use of pseudo-labels is the key practical concession. Without ground-truth labels, CARPRT must rely on the VLM's own predictions to partition images by class, which introduces noise, particularly for datasets where the base VLM performs poorly. The authors acknowledge this and examine filtering strategies (confidence-based and entropy-based), finding only marginal gains from explicit filtering. Iterative refinement, where weights are re-estimated using predictions from the previous iteration, yields steadier improvements.
The temperature hyperparameter controls how peaked the resulting weight distributions are. A temperature of 1.0 is found to be stable across tasks, which is practically convenient, though the paper notes that finding a universally optimal value remains difficult.
Results
Experiments span eleven classification benchmarks (Caltech101, DTD, EuroSAT, Aircraft, Food101, Flowers102, Pets, Cars, SUN397, UCF101, ImageNet) using three model configurations: CLIP with ViT-B/16 and ResNet50 backbones, and DeCLIP with ViT-B/32. The prompt pool consists of 247 templates from Allingham et al. (2023), held fixed across all methods.
Key findings:
- CARPRT achieves the highest average accuracy across all backbone and architecture configurations, with average gains of roughly 2-3 percentage points over WPE.
- Gains are largest on datasets with semantically well-separated categories: Flowers102 (+4.76% over WPE with CLIP ViT-B/16) and Pets (+6.75%).
- On Aircraft, gains are modest, which the authors attribute to weak initial pseudo-label quality and the mismatch between generic prompt templates and fine-grained aeronautical distinctions.
- CARPRT surpasses human-selected prompt subsets on several benchmarks, which is a meaningful result given that human selection introduces external domain knowledge.
- Under distribution shift (ImageNet variants: -A, -R, -Sketch, -V2), weights estimated on in-distribution ImageNet transfer well, with CARPRT maintaining its lead over MPE and WPE across all variants.
The ablation against CARPRT-Uniform (which averages the class-specific weights back to a global vector) isolates the contribution of class-specificity itself, showing an average gain of 2.39% from retaining per-class weights. This cleanly attributes the improvement to the class-aware design rather than to CARPRT's scoring mechanism alone.
Limitations and Broader Implications
The method's dependence on pseudo-label quality is its most significant practical constraint. For domains where CLIP-style models are already weak (specialised industrial or medical imagery, for instance), the pseudo-labels used to partition the reference set will be noisy, and the class-specific weight estimates will inherit that noise. The paper's finding that explicit filtering does not reliably help suggests this is a genuine ceiling rather than an easily engineered problem.
The computational overhead relative to WPE is non-trivial in principle: CARPRT must compute scores for all prompt-class combinations across the reference set, which scales as O(m * n * C) where m is the number of reference images, n is the number of prompts, and C is the number of classes. For large class spaces (ImageNet has 1000 classes) with large prompt pools (247 templates), this is a significant matrix computation, though it remains feasible given that it requires only forward passes.
The authors also explore CARPRT as a plug-in component in broader VLM adaptation settings: test-time adaptation, soft prompt tuning, LLM-generated prompt augmentation, and visual proxy learning. The consistent gains across these settings suggest the class-aware reweighting principle is not narrowly specific to the zero-shot ensembling problem.
One open question the paper does not fully address is how CARPRT behaves when the number of reference images per pseudo-class is very small, either because the dataset is small or because the class distribution is heavily imbalanced. Weight estimates derived from few images are likely to be unreliable, and understanding this failure mode quantitatively would be useful for practitioners.
Overall, CARPRT makes a clear and well-supported case that class-agnostic prompt weighting leaves systematic performance on the table. The theoretical framing is genuinely useful, the empirical evaluation is thorough across architectures and distribution conditions, and the method is simple enough to be a practical drop-in for any system that already performs prompt ensembling. The code is available at https://github.com/tmlr-group/CARPRT.