Most progress in language modelling over the past several years has come from scaling: more parameters, more data, more compute. A smaller but persistent research thread asks whether principled structural priors can substitute for raw scale. The Cognitive Categorical Transformer (CCT) is a serious attempt to answer that question empirically, at a scale large enough to be meaningful and under a methodological framework tight enough to be credible.
The paper's central claim is that a 306M-parameter model augmenting GPT-2 Small with components drawn from category theory and cognitive science reaches 21.27 validation perplexity on WikiText-103, compared with 24.19 for an identically fine-tuned GPT-2 Small baseline. That 2.92 PPL gap (12% relative) is attributed to the architectural augmentation rather than to data or compute advantages, because both runs share the same backbone initialisation, corpus, optimiser, batch size, sequence length, and 215,000-step budget. The only variable is what is wired in.
What the Architecture Actually Does
CCT wraps a pretrained GPT-2 Small backbone (124M parameters) with five additional parameter-bearing modules totalling roughly 182M parameters. The largest of these is GT-Full, a per-layer simplicial message passing component (~82M parameters across 12 layers) adapted from Bodnar et al.'s cellular sheaf work. Standard attention operates on pairwise token relationships; simplicial message passing propagates information across higher-order simplices (triangles, tetrahedra and so on), capturing combinatorial structure that dot-product attention cannot express.
The remaining modules are smaller and conceptually distinct. HierarchicalMemory (~48M) provides a three-tier external memory motivated by Self-Determination Theory. PrecisionWeightedPP (~14.2M) implements a predictive processing module with learned per-feature precision networks, loosely inspired by Friston's free energy framework. CausalAttention (~9M) and YonedaSelfModel (~3M) round out the parameter budget, the latter maintaining probe embeddings that approximate the Yoneda functor from category theory with a narrative GRU tracking temporal state.
The seven-phase activation schedule, in which components are progressively wired in during training, is an unusual design choice. It complicates direct comparison but is motivated by the difficulty of training all components simultaneously from a pretrained initialisation.
The Ablation That Matters
The paper's most important methodological contribution is the distinction between two ablation protocols, and the authors are admirably explicit about why this matters.
An eval-only ablation loads a fully trained CCT checkpoint and bypasses a component at inference time. This measures how much the trained model has come to depend on that component's output distribution. A retrain-from-scratch ablation trains a new model from the same initialisation with the component disabled throughout the full training run. This measures the architectural contribution of the component.
These are not the same quantity, and conflating them is a genuine methodological hazard in the ablation literature. The paper demonstrates the difference concretely: the eval-only protocol attributes 91.6% of measured benefit to GT-Full, while the retrain ablation attributes 84%. The 1.24x overstatement from eval-only ablation is a useful calibration point for anyone designing ablation studies.
The retrain result (E2) reaches 23.72 PPL with GT-Full disabled throughout, placing it 2.45 PPL above the full CCT and only 0.47 PPL below the fine-tuned baseline. GT-Full therefore accounts for 84% of the architectural improvement; the remaining non-GT-Full components collectively contribute only 0.47 PPL.
The Structure/Consistency Distinction
The paper's most conceptually interesting finding is what it calls the structure/consistency distinction. Three categorical consistency priors all failed to improve perplexity:
- Sheaf smoothing: enforcing local-to-global consistency across the residual stream
- Adjunction round-trip regularisation: penalising deviation from a left-right adjoint identity
- Curvature regularisation: penalising high discrete curvature in the attention graph
Two structural priors succeeded: simplicial topology (GT-Full) and precision-weighted prediction (PrecisionWeightedPP). The authors ground the sheaf failure independently by citing Bosca and Ghrist's result that feedforward ReLU forward passes already achieve minimal sheaf discrepancy, making the sheaf smoothing loss redundant by construction. Whether the same logic extends to the adjunction and curvature priors is less clear, but the empirical pattern is consistent.
The hypothesis that PrecisionWeightedPP's contribution is conditioned on GT-Full's structural sparsification is interesting but rests on a single paired observation. The E2 ablation shows only 0.07 PPL improvement from PP in the final phase, versus 1.40 PPL in the full CCT run. This could reflect genuine architectural conditioning, or it could be seed-level noise. The paper correctly identifies this as unverified and names the required counterfactual experiment.
Limitations and Open Questions
The authors are unusually candid about what this paper does not establish, which is worth taking seriously rather than treating as boilerplate.
All results are single-seed (seed 42). The headline numbers, 21.27, 24.19, and 23.72, are each one training run. Multi-seed replication is identified as a P1 priority and is absent from this revision. Given that the non-GT-Full contribution is only 0.47 PPL, the question of whether individual components within that group have real contributions or are noise cannot be answered without per-component isolation ablations across multiple seeds.
Downstream benchmarks (ARC-Easy, HellaSwag, BLiMP, LAMBADA, COPA) were measured against the RC2 checkpoint and have not been re-evaluated against the matched E1 or E2 baselines. This means the architectural improvement is validated only on WikiText-103 perplexity. Whether the GT-Full contribution transfers to downstream task performance, or whether it is absorbed by the perplexity metric specifically, remains genuinely open.
Scale is a third limitation. Everything here is at 306M parameters on a single corpus. The structure/consistency distinction, the 84% retrain share, and the conditional-PP hypothesis are all single-scale, single-dataset observations. The comparison to GPT-2 Large (22.05 PPL zero-shot, 774M parameters) is presented carefully as an external published reference rather than an architectural benchmark, which is the right framing, but it does highlight that CCT at 306M parameters is approaching GPT-2 Large territory on this metric. Whether that reflects genuine architectural efficiency or the specific properties of WikiText-103 fine-tuning is hard to disentangle.
The warmup-step heterogeneity across phases is a minor but real wrinkle in the matched-schedule claim. The practical effect is small across 215,000 steps, but the paper is right to flag it.
Despite these caveats, the CCT paper makes a genuine empirical contribution. The retrain-from-scratch validation of simplicial message passing at the 306M scale is the first of its kind on WikiText-103, and the methodological discussion of eval-only versus retrain ablations is valuable independent of the specific architecture. The structure/consistency distinction, if it replicates across seeds and scales, would be a meaningful organising principle for future work on categorical priors in neural networks. The natural next experiments are clear: multi-seed replication, per-component isolation ablations for the non-GT-Full modules, the GT-Full-without-PP counterfactual, and downstream benchmark re-evaluation against the matched baselines.