The shift toward disaggregated LLM inference, where prefill and decode phases run on separate GPU pools, has introduced a data movement problem that the field has largely treated as an afterthought. Existing systems like DistServe, Splitwise, and Mooncake transfer KV caches between GPU pools using uniform RDMA, ignoring the fact that bandwidth between two GPUs varies by up to 72x depending on their physical relationship. This paper, available on arXiv, presents TopKV, a topology-aware transfer orchestrator designed to close that gap.
The motivation is straightforward once you look at the numbers. For Llama-3-70B with a 4K-token prompt, the KV cache is 2.6 GB per request. At 100 requests per second, aggregate transfer demand hits 260 GB/s. Over InfiniBand RDMA at 25 GB/s effective throughput, a single transfer takes around 104 ms, which consumes more than a quarter of a 400 ms time-to-first-token budget. Via NVLink at 450 GB/s, the same transfer completes in 5.8 ms. The hardware bandwidth hierarchy is not subtle, and the cost of ignoring it is measurable in user-facing latency.
Key Contributions
TopKV makes three distinct technical contributions, each targeting a different aspect of the transfer problem:
- Topology-aware transport selection. The system discovers interconnect hierarchy at startup, cataloguing NVLink domain membership, NVSwitch fabric connectivity, PCIe hierarchy, and RDMA availability. For each transfer, it selects the optimal transport mode from five options: NVLink, NVSwitch, PCIe, RDMA, and TCP. This is the core insight: the correct transport depends on the physical relationship between source and destination GPUs, which changes with every request.
- Pipelined layer-by-layer transfer. Rather than waiting for prefill to complete before initiating transfer, TopKV begins sending layer l's KV cache while layers l+1 through L are still computing. For Llama-3-70B with 80 layers, remaining prefill compute after the first layer is approximately 39.5 ms. This hides 38% of RDMA transfer latency and 76% of PCIe transfer latency behind computation. For NVLink, the transfer completes entirely within the compute window.
- CXL 3.0 as a KV cache overflow tier. The paper models CXL 3.0 memory expanders as a shared overflow tier between GPU HBM and NVMe, offering 6x the capacity of HBM at 86x lower latency than NVMe. This addresses the memory capacity constraint that arises when long-context requests or high concurrency exhaust on-device HBM.
A fourth contribution, less central but worth noting, is the co-optimisation of MoE expert dispatch with KV cache placement. DeepSeek-V3's 256 routed experts, with 8 active per token, create a situation where routing decisions made for expert locality can inadvertently force cross-rack KV transfers. TopKV's placement logic attempts to account for both simultaneously, though the paper is candid that this remains analytically complex.
Methodology and Analytical Models
This is where the paper requires careful reading. TopKV is not evaluated on real hardware. The authors explicitly state that full evaluation requires multi-node clusters with heterogeneous interconnects and CXL 3.0 hardware that is beyond academic resources and not yet available in GPU clouds. What they present instead is a combination of analytical bandwidth models, component-level implementations, and projected analysis across three architectures.
The transfer latency model is simple: T = S/B + L, where S is cache size, B is transport bandwidth from published specifications, and L is per-transfer setup latency. The pipelining overlap model is slightly more interesting, computing effective transfer time as the maximum of the last-layer compute time and the residual transfer time after compute overlap. Both models are grounded in published hardware specifications rather than empirical measurement, which is a significant caveat.
The component implementations, including the KV Relay Orchestrator, Transfer Manager, Topology Manager, and Adaptive Decoder Pool, are described in enough detail to be credible as engineering work. The BandwidthThrottledSink wrappers that simulate different transport modes are a reasonable proxy for component-level testing, but they cannot capture the interference effects, contention, and protocol overhead that dominate real datacenter measurements.
Results and Projections
The headline numbers are 3 to 18x transfer latency reduction over uniform RDMA, depending on the physical relationship between source and destination. The 18x figure applies when source and destination share an NVLink domain, which the authors note occurs for 87.5% of GPU pairs on the same node in an 8-node, 64-GPU cluster. The 3x figure applies to cross-node transfers where PCIe rather than RDMA becomes the bottleneck.
The MLA analysis is a useful corrective to the main narrative. DeepSeek-V3's Multi-head Latent Attention compresses the KV cache to 250 MB, a 32x reduction versus equivalent MHA. At that size, even RDMA transfer takes around 10 ms, and NVLink reduces this to 0.6 ms. The absolute gains are smaller, and the authors acknowledge that topology-aware transport is less critical for MLA models, though still beneficial at high concurrency.
Limitations and Implications
The absence of empirical evaluation is the paper's most significant weakness, and the authors deserve credit for stating it plainly rather than obscuring it. Analytical models built on published specifications tend to be optimistic. Real NVLink bandwidth is rarely 450 GB/s in practice due to protocol overhead and contention. RDMA latency varies with fabric load. CXL 3.0 latency characteristics in production are not yet well-characterised in the literature. The projected improvements may hold directionally, but the precise multipliers should be treated as upper bounds.
There is also a scheduling complexity that the paper underweights. Topology-aware placement means the orchestrator must jointly consider request routing, GPU availability, and physical topology for every incoming request. At 100 requests per second, this scheduling problem becomes non-trivial, particularly for MoE models where expert placement interacts with KV cache locality. The paper describes the Adaptive Decoder Pool's role in dynamic role conversion but does not model the scheduling overhead.
That said, the core observation is correct and important. The field has built disaggregated inference systems that treat the network as a uniform resource, and it is not. As KV cache sizes grow with context length, and as production deployments push toward higher concurrency, the cost of topology-blind transfer will compound. The 72x bandwidth hierarchy in modern GPU clusters is a first-class architectural feature, not a footnote.
TopKV's contribution is primarily as a design document and analytical framework. It identifies the right problem, proposes a coherent architecture, and provides models that can guide hardware procurement and system design decisions even before empirical validation is possible. Whether the specific numbers hold will require access to the hardware the authors describe. But the framing, that KV cache transfer is a topology problem and should be treated as one, is likely to influence how the next generation of disaggregated inference systems are built.
The full paper is available at arXiv:2607.28633.