Most work on low-precision arithmetic for language models focuses on how faithfully individual values are represented. The product structure that emerges when those values are multiplied together is treated as a secondary concern, something to be handled by a conventional integer or floating-point multiplier sitting downstream of the quantization logic. A new paper on ArXiv, CurveFP: Rational-Radix Logarithmic Datatypes with Closed Products for Language Models, argues this separation is a missed opportunity and proposes a format family designed so that the representation geometry and the multiplication algebra are specified jointly from the outset.
The practical stakes are real. FP8 training is now standard at scale, and microscaling formats have reduced the overhead of per-block range adaptation. Yet in all these schemes, a nonzero product of two quantized values does not generally land on another quantized value. The accumulation path must therefore bridge two different numeric worlds. CurveFP attempts to close that gap at the format level rather than patching it in the accumulator.
What CurveFP Actually Does
The core idea is to distribute quantized magnitudes across K interleaved logarithmic curves rather than a single uniform or floating-point grid. Each nonzero element carries a sign bit, a two's-complement exponent, and a curve index, all sitting under a shared power-of-two block scale. The crucial design choice is that curve indices are spaced uniformly. This means that when two elements are multiplied, the result is another element in the same codebook: signs combine by XOR, exponents add with a carry, and curve indices add modulo K. No lookup table is needed, no projection back onto the grid, and no general multiplier.
The rational radix, written as 2 to the power p/q, is the second key parameter. Choosing a non-integer exponent base lets the designer tune how dynamic range trades against local resolution. Coarser logarithmic formats based on integer powers of two have long been known to sacrifice local precision; the rational radix recovers some of that resolution without abandoning the additive multiplication property. A derived quantity, H = qK / gcd(p, qK), counts the number of distinct fractional exponent classes, which the authors call phases. This phase count determines exactly how many distinct accumulation bins are needed, making the accumulation cost an explicit function of the format parameters rather than an implementation detail.
Two concrete instantiations are proposed. CurveFP8 uses an E4C3 layout for forward weights and activations and E5C2 for backward gradients, deliberately mirroring the role-specific exponent allocation of FP8 E4M3/E5M2. CurveFP7 uses E3C3 to shave one bit per element for deployment, reaching 7.008 effective bits per weight-activation pair at a block size of 512, while retaining eight product phases.
Methodology and Experimental Design
The evaluation is structured around three axes that together cover the model lifecycle. Post-training quantization on Llama-3-8B, Qwen3-8B, Qwen3.5-9B, and Falcon-H1-7B tests inference quality across architectures. Backward-GEMM diagnostics measure normalised mean squared error for all three matrix products in a training step: forward, activation gradient, and weight gradient. Matched from-scratch pretraining runs three 128.3M-parameter models, one each in BF16, FP8, and CurveFP8, for 3B tokens per seed across three seeds, followed by WikiText-103 and PG-19 evaluation and a suite of downstream tasks.
The controls are careful. All formats share identical GEMM kernels and evaluation code. Every linear layer except the tied output vocabulary head is quantized; normalisation and nonlinear operations remain in BF16. All reported numbers come from saved JSON artifacts, which is a sensible hedge against evaluation-code drift across long experimental runs.
One methodological point worth flagging: the paper compares CurveFP7 against tensorwise FP8 rather than against block-scaled FP8. Tensorwise FP8 is a weaker baseline for inference because it lacks the per-block range adaptation that microscaling formats provide. The more relevant comparison for deployment would be block-scaled FP8 or MX formats at comparable bit widths, and the paper does include some of this but the headline inference numbers use the tensorwise baseline.
Results
The key findings, taken at face value, are encouraging:
- CurveFP7 beats tensorwise FP8 perplexity on all four 7B-9B models while using one fewer bit per element, and stays within 1.32% of native BF16/FP16 quality.
- CurveFP8 achieves lower operand NMSE than FP8 in all 36 paired forward and backward GEMM comparisons across training windows.
- In matched 3B-token pretraining, CurveFP8 reaches mean BF16-inference perplexity 22.5366 versus 22.5407 for FP8. All nine training lanes (three formats times three seeds) complete without divergence.
- The trained CurveFP8 checkpoints improve WikiText-103 perplexity relative to FP8 checkpoints in all 12 seed-format comparisons.
- PG-19 and downstream task results show mixed outcomes, which the authors interpret as transfer parity rather than universal superiority. This is an honest reading of the data.
The perplexity gap between CurveFP8 and FP8 after pretraining is small: roughly 0.004 points on a scale of 22.5. The paper is transparent that this supports parity rather than a clear win, and the seed variation is consistent with that interpretation. The format-induced penalty metric (the gap between low-precision and BF16 inference perplexity) does favour CurveFP8 in all three seeds, which is a cleaner signal of format quality than absolute perplexity.
Limitations and Open Questions
Several questions are left open. The experiments top out at 128.3M parameters for training, which is substantially smaller than the 7B-9B models used for inference evaluation. Whether the closed-product arithmetic advantage survives at larger training scales, where gradient noise and optimizer state interact with quantization error in more complex ways, is not established.
Hardware realisability is discussed in terms of index adders and phase reduction, but no silicon area or latency numbers are provided. The algebraic simplicity of sign XOR plus integer addition is appealing, but the practical benefit depends on whether the phase accumulation structure maps cleanly onto existing tensor core designs or requires custom datapath logic. This is the central unanswered engineering question for anyone considering adoption.
The rational radix idea is theoretically elegant but introduces a hyperparameter, the p/q ratio, that must be chosen per deployment context. The ablations cover several configurations, though the sensitivity of downstream quality to this choice across different model families and task distributions remains underexplored.
Taken together, CurveFP is a coherent piece of arithmetic co-design that addresses a real gap in how low-precision formats are specified. The algebraic closure result is clean, the phase count formula gives a principled handle on accumulation complexity, and the empirical results are competitive with FP8 at reduced bit width. The open questions around hardware cost and large-scale training are significant, but they are the right questions to ask next rather than objections to the approach itself.