5 min read
The question of where large language model inference actually runs is no longer purely academic. As consumer-grade hardware closes the gap with data centre GPUs for specific workload profiles, the engineering tradeoffs involved in local deployment deserve careful scrutiny. A recent post on jola.dev covering local model execution on the M4 with 24GB unified memory generated substantial discussion on Hacker News, and this analysis thinks it surfaces some genuinely important architectural considerations that are worth unpacking properly.
The central technical fact underpinning all of this is Apple's unified memory architecture. On a discrete GPU system, you have two separate memory pools: system DRAM and VRAM. Moving tensors between them is expensive, both in latency and bandwidth. The M4's unified memory eliminates that boundary entirely. The CPU, GPU, and Neural Engine all address the same physical memory pool, which means a 24GB configuration can dedicate the full allocation to model weights without the overhead of host-to-device transfers.
the author would point out that this matters most at inference time, not training. During autoregressive generation, the dominant cost is memory bandwidth, not raw floating-point throughput. Each forward pass must load the full KV cache and model weights from memory. The M4's memory bandwidth sits around 120 GB/s for the base configuration, which is substantially lower than an H100's 3.35 TB/s, but the absence of PCIe transfer overhead and the tight integration with the Neural Engine makes the effective utilisation considerably higher for batch-size-one interactive inference.
This is the regime that matters for local deployment. You are not running batch-64 inference on a MacBook. You are running a single user query, and for that specific workload shape, the M4 is surprisingly competitive.
The jola.dev post explores running several model families at various quantisation levels, which is where the author finds the most analytically interesting material. The practical ceiling of 24GB unified memory constrains which models are viable at which precision levels. A rough accounting:
The implication is that 24GB is a meaningful but not transformative threshold. You can run genuinely capable models, particularly in the 7B to 13B range where recent training improvements have dramatically improved capability per parameter. Models like Mistral 7B, Llama 3.1 8B, and Qwen2.5 14B are all accessible at this memory capacity. What you cannot do is run frontier-scale models locally, even with aggressive quantisation.
Notes that the quantisation methodology matters as much as the bit-width. GGUF format with llama.cpp and Apple's MLX framework both support various quantisation schemes, and the quality degradation curves are not identical across model families. Q4_K_M tends to preserve perplexity better than naive Q4_0 for most architectures, and the difference is measurable on reasoning benchmarks. Treating all 4-bit quantisation as equivalent is a methodological error that practitioners should avoid.
The post reports token generation speeds in the range of 20-50 tokens per second for smaller models, which reads as impressive but requires contextualisation. Human reading speed is approximately 4-5 tokens per second for comfortable comprehension. So 20 tok/s is already well above the threshold for interactive use. The practical question is whether the model is capable enough to be useful, not whether it is fast enough.
For the author, the more interesting metric is time-to-first-token, which the post does not address in depth. Prefill latency scales with prompt length and is compute-bound rather than memory-bandwidth-bound. Long system prompts or retrieval-augmented contexts can produce noticeable prefill delays even on hardware that generates tokens quickly. This is a known limitation of the autoregressive architecture that affects all inference hardware, but it is worth flagging for users who plan to use local models with substantial context injection.
The Hacker News discussion thread raised several points about thermal throttling during sustained inference sessions. The M4's thermal envelope is constrained by the fanless or limited-fan designs of many configurations. Extended generation at peak throughput will trigger throttling, and sustained token rates over a multi-hour session will be lower than peak benchmarks suggest. This is not a criticism specific to Apple silicon; it is a physical constraint of the form factor.
Beyond raw performance, Thinks the architectural argument for local inference deserves serious treatment rather than being dismissed as a hobbyist preference. There are at least three distinct motivations that stand up to scrutiny:
The counterargument is model capability. GPT-4o and Claude Sonnet class models remain substantially more capable than anything runnable locally on 24GB of unified memory. For tasks requiring genuine reasoning depth or broad world knowledge, the capability gap is real. the author would characterise the current state as: local models are good enough for a large fraction of practical tasks, but not all of them, and knowing which is which requires careful task analysis rather than blanket assumptions in either direction.
Stepping back, the trajectory here is clear. Each generation of Apple silicon has increased the viable model size for local inference. The M1 with 16GB could run 7B models awkwardly. The M2 and M3 improved throughput. The M4 with 24GB runs 13B models comfortably and 34B models at the edge of viability. The M5 generation will presumably push this further.
This progression is not happening in isolation. Quantisation research is improving the capability-per-bit tradeoff. Model distillation is producing smaller models with capabilities that would have required much larger models two years ago. The combination of better hardware and better models means the local inference capability curve is steeper than the hardware improvement curve alone would suggest.
the author sees this as relevant to the broader research agenda around model efficiency and deployment. Work on speculative decoding, continuous batching, and adaptive quantisation all have implications for edge deployment that extend beyond data centre optimisation. The M4 results are a useful empirical data point in that research context, even if the post itself does not frame them that way.
For researchers and engineers thinking seriously about on-device AI, the jola.dev post is worth reading as a practical benchmark reference. It does not resolve the fundamental capability tradeoffs, but it provides honest throughput numbers and a clear account of what is and is not feasible at the 24GB memory tier. That kind of empirical grounding is more useful than theoretical projections.
the author will continue tracking developments in local inference efficiency and edge deployment architectures. If you are working in this area and want to discuss methodology or share results, the contact page is the best place to reach out. Further context on This research interests is available on the about page and research page.