As AI datacenters cycle through hardware generations at an accelerating pace, enormous quantities of still-functional GPUs flow into secondary markets. The question of what happens to these accelerators, whether they can be productively repurposed rather than scrapped, sits at the intersection of economics, environmental policy, and systems engineering. This paper, DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs, provides the most thorough empirical answer to that question to date. The authors did not simulate a hypothetical cluster; they built one from scratch, ran it in production for a year, and measured everything.
What Was Built and Why It Matters
The team assembled a 128-GPU cluster using entirely second-hand components: NVIDIA V100 GPUs, CPUs, motherboards, and DRAM, all sourced from the secondary market. The total GPU acquisition cost came to approximately $7,680, compared to $600,000 for an 8-GPU B200 node capable of running the same 70B-parameter models. That is not a rounding error; it is a 78x difference in capital expenditure.
The motivation is not merely frugality. The paper frames the problem around three overlapping pressures on the AI infrastructure ecosystem: the embodied carbon cost of manufacturing new accelerators, supply-chain concentration among a small number of vendors and foundries, and the structural inaccessibility of frontier hardware for most research institutions and smaller organisations. Second-hand GPUs, the argument goes, address all three simultaneously, provided the deployment conditions are right.
To make older hardware viable for modern LLM serving, the authors developed a custom inference engine in Rust implementing a pipeline-parallel strategy. Unlike tensor parallelism, which requires high-bandwidth GPU-to-GPU interconnects (NVLink or equivalent), pipeline parallelism distributes model layers across devices and tolerates slower interconnects. This is a sensible engineering choice for V100s, which lack the NVLink bandwidth of more recent hardware and have limited HBM capacity (16GB or 32GB per card). The result is that throughput scales roughly linearly with device count, which is a meaningful property when you have 128 of them.
Performance and Cost Results
The headline throughput numbers are striking. For LLaMA 3.1-8B under a prefill-heavy workload, the 128-V100 pod achieves approximately 224,000 tokens per second, compared to 62,651 for a single B200. For 70B, the comparison is tighter: 1,512 tokens per second for the V100 pod versus 37,045 for an 8-GPU B200 system. The V100 cluster achieves roughly 3.2x the cost-effectiveness of a B200 system on 70B inference when measured as throughput per dollar of capital expenditure.
The total cost of ownership analysis is where things get more nuanced. The paper models 3-year, 5-year, and 8-year deployment lifetimes across four energy mixes:
- US grid average (20% renewable): ~$0.12/kWh
- China grid average (30% renewable): lower unit cost
- Brazil grid (90% renewable): moderate cost
- Chinese wind farms (100% renewable): lowest carbon intensity
For the 8B model, the 128-V100 cluster is the cheapest option across all energy mixes and all deployment horizons considered. For the 70B model, the picture is more complicated. Under the China energy mix, the V100 pod's 3-year total cost per token is approximately 1.5x that of an H100 system. The capital cost advantage erodes as model size grows because larger models require more devices and more electricity per token, and V100s are simply less energy-efficient silicon than H100s or B200s. The pipeline-first strategy helps, but it cannot fully compensate for a two-generation gap in process node and memory bandwidth.
The Carbon Accounting Problem
This is where the paper makes its most important and underappreciated contribution. The environmental case for repurposing hardware is intuitively appealing: avoid manufacturing new chips, extend the useful life of existing ones, reduce e-waste. The authors take this intuition seriously and subject it to quantitative scrutiny, and the results are sobering.
Under grid-average carbon intensity (the US energy mix), the V100 cluster produces approximately 4x higher carbon emissions per token for 8B models and over 40x higher for 70B models, compared to a B200 system. The disparity for 70B is particularly severe because the model requires so many more V100s to serve, each drawing substantially more power per useful computation than a modern accelerator.
The paper handles embodied carbon with appropriate care, presenting two accounting scenarios. The first treats the V100s as having extended useful life, allocating a share of their original manufacturing carbon to the new deployment. The second assumes the devices would otherwise have been landfilled, zeroing out embodied carbon and crediting the deployment with avoided waste. Even under the more favourable zero-embodied assumption, the operational carbon dominates at scale, particularly for 70B models on carbon-intensive grids.
The conclusion is not that hardware repurposing is environmentally harmful per se. It is that the environmental benefit is conditional. When deployed on clean electricity (the Chinese wind farm scenario, or equivalent), carbon footprints become competitive and the case for second-hand hardware holds. When deployed on a typical grid, the operational emissions of older, less efficient hardware can dwarf the embodied carbon savings. This is a finding with real policy implications for how organisations and regulators should think about hardware lifecycle and data centre siting.
Limitations and Open Questions
The paper is admirably honest about its constraints. The V100 is a specific choice of second-hand hardware; results would differ for A100s or older Volta variants available at different price points. The custom Rust inference engine, while effective, is not directly comparable to vLLM in terms of software maturity, feature coverage, or batching sophistication. The authors acknowledge this and use vLLM for all baseline comparisons, which is methodologically clean, but it means the V100 results may not reflect the ceiling of what optimised software could achieve on that hardware.
There are also questions the paper does not fully address. Reliability and maintenance overhead for second-hand components are mentioned but not quantified in depth. Secondary market pricing is volatile; the $60-per-GPU figure reflects a specific market moment, and V100 prices have fluctuated considerably. The analysis also focuses on inference rather than fine-tuning or training, where memory capacity constraints on V100s would be far more limiting.
Perhaps the most interesting open question is how this analysis changes as the secondary market matures. If demand for second-hand V100s increases because papers like this validate their use, prices will rise and the economics shift. The authors' framework is the right one for thinking through these trade-offs, but the specific numbers will not remain static.
For researchers and engineers considering alternative infrastructure strategies, or anyone thinking seriously about the sustainability of AI at scale, this paper is worth reading in full. The methodology is sound, the data is real, and the conclusions are appropriately qualified. Read the full paper at arXiv:2608.14614.