Most of what we know about optimising diffusion model inference comes from the NVIDIA ecosystem. Quantisation, parallel inference, attention optimisations like Token Merging, and mixed-precision pipelines have all been developed and validated against CUDA hardware. The implicit assumption is that these techniques generalise. This paper, available at arXiv:2605.16259, tests that assumption directly on Apple Silicon and finds it largely wrong.
The practical motivation is real-time camera-based image-to-image transformation on the M3 Ultra, a machine with a 60-core GPU and 512 GB of unified memory. This is not a niche use case. Creative tools, live video processing, and interactive installations all benefit from low-latency generative inference, and a significant portion of professional creative hardware runs macOS. The absence of systematic benchmarking on this platform has left practitioners guessing.
What the Study Actually Did
The work is structured as a sequential ablation across ten optimisation phases, each building on or branching from the previous. The techniques explored include:
- CoreML model conversion
- INT8 and lower-bit quantisation
- Token Merging (ToMe)
- Apple Neural Engine (ANE) offloading
- Compact model substitution (exploring smaller architectures)
- Frame interpolation for temporal smoothing
- kNN search-based synthesis as an alternative generation path
- Pix2pix-turbo for single-step conditional generation
- Optical flow-based frame skipping
- Knowledge distillation, specifically the SDXS-512 model
Each phase was evaluated quantitatively on throughput, with the target metric being frames per second at 512x512 resolution for live camera input. This phased structure is methodologically sensible: it avoids conflating the effects of multiple simultaneous changes and gives a cleaner picture of what each technique contributes independently.
The Surprising Findings
The headline result is 22.7 FPS achieved by combining CoreML conversion of SDXS-512 with a three-thread camera pipeline. That is genuinely real-time for this class of task. But the more scientifically interesting findings are the failures.
Quantisation produced no meaningful speedup. On CUDA hardware, reducing precision from FP16 to INT8 typically yields substantial throughput gains because it reduces memory bandwidth pressure and allows more operations per clock cycle. On the M3 Ultra's unified memory architecture, this benefit does not materialise. The memory subsystem behaves differently enough that the usual arithmetic does not apply. This is not a minor footnote; it has direct implications for anyone porting a CUDA-optimised pipeline to Apple Silicon and expecting similar results.
Parallel inference was similarly ineffective. Running multiple inference threads simultaneously, which can saturate GPU compute on discrete graphics hardware, did not improve throughput on the M3 Ultra. The unified memory architecture appears to introduce contention or scheduling overhead that negates the parallelism gains.
The Apple Neural Engine results are also worth examining carefully. The ANE is a dedicated matrix-multiplication accelerator with impressive peak throughput on paper, and it is the obvious target for offloading transformer-based components. The paper reports that it is unsuitable for large-scale diffusion models. This likely reflects the ANE's architectural constraints around supported operation types and memory transfer overhead between the ANE and GPU, rather than raw compute capacity. The finding aligns with anecdotal developer reports but this appears to be the first systematic quantitative confirmation in the context of diffusion models.
What did work was model selection and CoreML conversion. SDXS-512 is a distillation-specialised model designed for single-step or very-low-step inference, and its compact computational graph converts cleanly to CoreML's execution model. The three-thread camera pipeline manages capture, preprocessing, and inference concurrently without the contention that afflicts parallel inference on the same model.
Methodological Considerations
The study is honest about its scope. It targets a single hardware configuration, one generation of Apple Silicon at the high end of the product line. The M3 Ultra's 60-core GPU and 512 GB unified memory pool is not representative of the M3 Pro or M2 MacBook Air that most users actually own. Generalising these findings across the Apple Silicon family requires caution; the memory bandwidth ratios and GPU core counts differ substantially between tiers.
The choice of img2img as the evaluation task also constrains the conclusions somewhat. Text-to-image generation has a different computational profile, particularly around the conditioning pathway, and the frame-to-frame coherence requirements of camera input justify some of the pipeline choices (optical flow, frame interpolation) that would not apply in a batch generation context.
There is also a question of what baseline is being compared against. The paper frames its contribution relative to CUDA optimisation wisdom, but direct numerical comparisons with equivalent NVIDIA hardware running optimised pipelines are not the focus. The contribution is the systematic characterisation of Apple Silicon behaviour, not a competitive benchmark.
Implications for the Field
The broader significance here is methodological rather than purely applied. The diffusion model optimisation literature has developed almost entirely on CUDA hardware, and the techniques that have become standard practice reflect CUDA's architectural properties: discrete memory hierarchies, high-bandwidth GPU memory, and well-understood quantisation paths via libraries like TensorRT and bitsandbytes. Apple Silicon's unified memory architecture breaks several of the assumptions that make these techniques effective.
For practitioners building on Apple Silicon, the practical takeaways are clear: invest in model selection and CoreML conversion before reaching for quantisation or parallelism. Distillation-specialised compact models appear to be the most productive direction. The ANE should be treated with scepticism for large transformer-based models until better tooling exists.
For researchers, this work opens a more fundamental question about how optimisation insights should be validated across hardware architectures. The field has implicitly assumed a degree of transferability that this study challenges. As inference increasingly moves to edge and consumer hardware with non-CUDA architectures, including mobile Apple Silicon, Qualcomm Snapdragon NPUs, and Intel integrated graphics, the need for platform-specific systematic evaluation becomes more pressing.
The paper is available in full at https://arxiv.org/abs/2605.16259. For anyone working on generative model deployment outside the CUDA ecosystem, it is a useful corrective to assumptions that have gone untested for too long.