Most benchmarks for LLM serving systems are built on traces that are either short-lived, sampled, or synthetically generated. This creates a significant gap between the conditions under which systems are evaluated and the conditions they actually face in production. A new paper from Nixon et al., A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing, attempts to close that gap by releasing and analysing a full one-year production trace from Chutes, a real-world LLM serving platform.
The timing matters. As LLM inference has shifted from a research curiosity to a critical cloud workload, the systems community has struggled with a reproducibility problem: you cannot design good serving infrastructure without understanding real traffic patterns, but real traffic patterns are rarely shared. This paper directly addresses that bottleneck.
What the Paper Contributes
The core contribution is both empirical and infrastructural. The authors provide:
- A one-year production trace covering many models and users, including both popular and long-tail models, which prior studies have largely ignored
- A multi-perspective analysis spanning aggregate, temporal, model-level, and user-level views of the workload
- A longitudinal study of how workload characteristics shift over time, something almost entirely absent from existing literature
- A public release of the full trace to support downstream research without reliance on synthetic proxies
The decision to release the full trace rather than a sample is significant. Sampled traces can obscure the long-tail behaviour that often drives worst-case system performance. If the trace is as complete as described, it should become a standard reference dataset for the serving systems community.
Methodology and Scope
The study draws on production data from Chutes, analysed across four analytical lenses. Aggregate characterisation establishes baseline statistics on request volumes, prompt and completion lengths, and inter-arrival times. Temporal analysis examines how these statistics shift across the year, capturing seasonal patterns, growth trends, and the emergence or retirement of specific models. Model-level analysis disaggregates traffic by model, which is important because serving behaviour differs substantially between a high-throughput popular model and a rarely-queried specialised one. User-level analysis examines how individual users interact with models over time, including patterns of model switching, query frequency, and session structure.
This four-axis decomposition is methodologically sound. Aggregate views of serving workloads are well understood to be misleading: they smooth over heterogeneity that directly affects decisions about batching strategies, KV cache sizing, and replica placement. By explicitly separating model-level and user-level views, the authors can surface structure that aggregate statistics would hide.
The focus on caching and load-balancing as downstream applications of the trace analysis is well chosen. KV cache reuse is one of the more practically impactful optimisations in LLM serving, and its effectiveness depends heavily on the prefix-sharing structure of real queries. Similarly, load-balancing decisions depend on understanding request heterogeneity across models and time. Both problems are poorly served by synthetic workloads.
Key Findings and Implications
Based on the paper's framing, several findings stand out as particularly consequential for the systems community:
- Workload evolution is substantial over a year. This is the central longitudinal finding. Traffic patterns, model popularity distributions, and request characteristics all shift meaningfully, which means systems tuned to early-period traces may be poorly matched to later traffic. Static benchmarks are not just incomplete; they may be actively misleading for capacity planning.
- Long-tail models carry non-trivial traffic. Focusing only on popular models misses a significant portion of real production load. This has direct implications for multi-model serving systems, which need to handle cold-start latency and memory pressure for infrequently-used models without degrading service for popular ones.
- User-model interaction structure is complex. Users do not simply issue independent queries. Session structure, repeated prefixes, and model-switching behaviour all affect how caching and scheduling should be designed. The user-level analysis appears to surface patterns that aggregate request logs would entirely obscure.
- Temporal patterns affect both caching and load-balancing. Diurnal and weekly cycles in request volume affect optimal replica counts and cache eviction policies. A serving system that ignores temporal structure will either over-provision during quiet periods or under-serve during peaks.
Limitations and Open Questions
The paper's primary limitation is one it shares with all single-platform trace studies: generalisability. Chutes is one serving environment with a particular user base, model selection, and business context. Traffic patterns on a consumer-facing chatbot platform, an enterprise API gateway, or a research cluster would likely differ substantially. The authors should be commended for releasing the trace, but the community will need multiple traces from diverse deployment contexts before strong generalisations can be drawn.
There is also a question of what the trace captures and what it does not. Production traces typically record request-level metadata but not the content of prompts and completions, for obvious privacy reasons. This limits the ability to study semantic patterns in queries, which are relevant for understanding prefix-sharing opportunities and model selection behaviour. The authors appear to be working within these constraints, but it is worth being explicit about what analyses are foreclosed.
The focus on caching and load-balancing, while well motivated, leaves other important serving problems underexplored. Autoscaling, disaggregated prefill-decode architectures, and speculative decoding all have workload dependencies that this trace could potentially inform. Releasing the full trace is the right move here; it allows the community to apply the data to problems the original authors did not anticipate.
Finally, the paper was submitted in mid-2026, meaning the trace covers a period of rapid change in the LLM ecosystem. Model sizes, serving frameworks, and user expectations all shifted considerably during this window. Interpreting the longitudinal findings requires some care about which changes reflect fundamental workload dynamics versus artefacts of a particularly turbulent period in the technology's development.
Overall, this is a valuable contribution. The field has needed realistic, long-horizon production traces for some time, and the combination of rigorous analysis with a full data release makes this paper worth close attention for anyone working on LLM serving infrastructure. The full paper is available at arXiv:2608.13573.