Most research into jailbreaking vision-language models (VLMs) has focused on what is depicted in an adversarial image: embedded typographic text, adversarially optimised objects, or harmful visual content smuggled past content filters. A new paper from arXiv, Adversarial Style Optimization (ASO), takes a different angle entirely, asking whether the style of an image, independent of its semantic content, can systematically undermine a model's safety alignment. The answer, backed by experiments across GPT-4.1-mini, Gemini-2.5-Flash, Qwen3-VL, and LLaVA-OV-1.5, is yes, and the effect is consistent enough to be practically concerning.
The Core Observation
The paper's central empirical finding is what the authors call Stylistic Inconsistency: VLMs can understand image content reliably across a wide range of visual styles, yet their safety mechanisms are disproportionately sensitive to those same stylistic variations. A model that correctly refuses a harmful request when presented with a photorealistic adversarial image may comply when that same image is rendered as a pencil sketch or vintage photograph. The content has not changed; the style has.
This is not an entirely surprising observation given what we know about how safety fine-tuning works. Alignment training tends to operate on the distribution of inputs the model was trained to refuse, and if that distribution skews toward photorealistic or naturalistic imagery, stylistic outliers may simply fall outside the learned refusal boundary. What the paper contributes is a systematic framework for exploiting this gap rather than stumbling across it by accident.
Methodology: Two-Stage Probe and Optimise
The ASO framework operates in two sequential phases. The first is a Style Sensitivity Probing stage, where a pool of off-the-shelf visual styles (pencil sketch, vintage photo, and similar filters) is applied to existing adversarial images and evaluated against the target VLM. This identifies which style category produces the highest baseline Attack Success Rate (ASR) uplift without any optimisation.
The second phase is where the technical novelty lies. Rather than using the best-performing style naively, ASO fine-tunes an image-editing model (specifically FLUX-Kontext) using Group Relative Policy Optimization (GRPO) to discover the optimal parameters within that style. The intuition is that a generic pencil sketch filter has many degrees of freedom, such as stroke density, line weight, and tonal mapping, and the adversarially optimal configuration within that space is unlikely to be the default one. GRPO, which normalises rewards across groups of sampled outputs rather than requiring a baseline value function, is well-suited to this kind of sparse, high-dimensional optimisation problem.
The reward signal guiding this RL process is a Structurally-Tiered Reward Function that combines two components. A cheap logit-based signal detects explicit refusals by examining the model's token probabilities directly. A more expensive semantic evaluation from a judge model (HarmBench) provides high-fidelity assessment of whether the response is genuinely harmful. Combining these allows the agent to get useful gradient signal even when full judge evaluations would be prohibitively slow at scale.
The plug-and-play framing is important here. ASO is not a standalone attack; it is an enhancement layer that wraps around existing attacks. The paper evaluates it on top of FigStep, QR Attack, SI Attack, and HIMRD, treating those as base attacks and measuring the ASR delta after applying ASO.
Results
The experimental results are reported across two benchmarks, MM-SafetyBench and VLBreakBench, and four models. Key findings include:
- On Gemini-2.5-Flash, applying ASO to QR Attack increases ASR from 55.04% to 62.79%, a gain of nearly 8 percentage points.
- On LLaVA-OV-1.5, ASO applied to SI Attack improves ASR from 37.82% to 44.25%.
- Even HIMRD, which already achieves 87.38% ASR on Qwen3-VL, sees a further gain to 89.52% under ASO.
- Across the 13 fine-grained MM-SafetyBench categories, ASO consistently increases both ASR and Harmfulness Score, including in near-saturated categories where ASR gains are marginal but the semantic severity of successful jailbreaks increases.
- On VLBreakBench, IDEATOR combined with ASO reaches 53.27% on Qwen3-VL and 75.54% on GPT-4.1-mini.
The Harmfulness Score metric deserves attention here. It is defined as the log-probability difference between the judge model assigning "yes" versus "no" to a harmful response. The fact that ASO increases this score even in saturated categories suggests the optimised style is not just crossing a binary threshold more often; it is producing responses that are more unambiguously harmful according to the judge. That is a qualitatively different and more troubling result than a simple ASR increase.
Limitations and Open Questions
Several limitations are worth flagging. The probing phase relies on a fixed pool of style categories, and the choice of which styles to include could meaningfully affect which vulnerabilities are identified. The paper does not discuss how sensitive the final ASR gains are to this initial pool selection, nor whether the optimal style discovered for one target model transfers to another without re-optimisation.
The use of FLUX-Kontext as the image-editing backbone is also a practical constraint. Fine-tuning a diffusion model via RL is computationally expensive, and the paper does not report training costs in detail. This matters for assessing whether ASO is a realistic threat in practice or primarily a research demonstration.
There is also a question about the generality of the Stylistic Inconsistency finding across model families. The experiments cover four models, which is a reasonable set, but all of them share broadly similar training pipelines. Models trained with more diverse stylistic augmentation during alignment, or with explicit style-invariant safety objectives, might not exhibit the same vulnerability. The paper does not address what a targeted defence would look like.
From a broader research perspective, this work raises an important question about the scope of safety evaluation. If style is an exploitable attack surface, then benchmarks that evaluate safety only on naturalistic or photorealistic inputs are systematically underestimating model vulnerability. The implication for red-teaming practice is that stylistic variation should be a standard part of the evaluation protocol, not an afterthought.
The code is available at github.com/bingjunluo/ASO, which should allow the community to reproduce and extend these findings. Whether the defensive community responds with style-augmented safety training or more principled approaches to style-invariant alignment remains to be seen, but this paper makes a credible case that the problem is real and currently underaddressed.