Large Language Models are increasingly deployed in financial services, yet a persistent and underappreciated failure mode undermines their reliability: arithmetic hallucination. A model can produce a fluent, confident, financially plausible answer that is simply numerically wrong. This is not a knowledge gap that retrieval-augmented generation can fix. It is an architectural problem. The paper introducing CIFQA (Calculation-Intensive Financial Query Answering) makes exactly this argument, and backs it with empirical evidence that is hard to dismiss. Read the full paper on arXiv.
The Problem: Architecture, Not Information
The authors begin from a clean diagnostic observation. Frontier models, specifically GPT-5.3, Gemini 3, and Claude Sonnet 4.6, were given fixed deposit queries along with complete rate cards, relevant formulas, and detailed benchmark instructions. They still failed systematically. The failure modes were not random; they clustered into identifiable categories: incorrect interest-rate slab selection from structured tables, mishandling of rolling-year denominators across leap-year boundaries, payout schedule reconstruction errors, and numerical drift across chained multi-step calculations.
This is a meaningful distinction. If the errors arose from missing information, RAG pipelines would be the natural remedy. But the persistence of errors under complete information conditions points to something more fundamental: probabilistic token prediction is not a reliable substrate for deterministic arithmetic execution. The paper formalises this as the CIFQA problem class, defined by exactness requirements, multi-step computation, structured data dependency, rule-based conditional logic, and temporal sensitivity. The framing is clean and the motivation is well-grounded in prior literature on LLM fragility in mathematical reasoning.
Methodology: Strict Separation of Concerns
The CIFQA framework enforces a hard boundary between what LLMs do and what deterministic engines do. LLM agents handle query routing, parameter extraction, computation planning, and response formulation. All arithmetic, including rate lookup, calendar-aware tenure computation, compounding, payout scheduling, and premature-withdrawal penalty logic, is delegated to Python-based deterministic tools. The LLM never touches a number in the computational sense; it only interprets and plans.
The pipeline has five stages:
- Routing: Classifies the query as computation-intensive, policy-oriented, or hybrid.
- Parameter extraction: Structures natural language inputs into typed financial parameters (principal, tenure, compounding frequency, customer category, etc.).
- Computation planning: Generates an execution plan specifying which deterministic tools to invoke and in what order.
- Deterministic execution: Python tools execute the plan with exact arithmetic, calendar-aware date handling, and rule application.
- Response generation: The LLM synthesises the deterministic outputs into a natural language answer.
The framework is instantiated on fixed deposit queries and evaluated on a curated benchmark of 126 queries spanning calculation-intensive scenarios, rate lookup tasks, policy interpretation, and edge cases. Ground truth for calculation-intensive queries was computed manually and verified against spreadsheet implementations, with correctness defined by a strict tolerance of plus or minus 1 INR. This is appropriately rigorous for the domain.
Results: Architecture Beats Scale
The headline result is striking. A 17B parameter open-source backbone (Llama-Scout-17B) operating within the CIFQA framework achieves 95.54% accuracy on calculation-intensive queries and 90.87% overall accuracy. This outperforms GPT-5.3 (45.05% on calculation-intensive queries), Gemini 3 (70.30%), and Claude Sonnet 4.6 (83.66%), all evaluated with complete financial context through their native conversational interfaces.
Category-wise results are particularly informative:
- CIFQA achieves 100% on quarterly payout, monthly payout, cumulative FD, TDS-related, half-yearly payout, and edge-case categories.
- Premature withdrawal queries reach 92.86%, compared to 78.57% for Claude Sonnet 4.6 and 35.71% for Gemini 3.
- The weakest categories for CIFQA are Rate Analytics (58.33%) and RAG Rules (72.00%), where the task shifts toward higher-order pattern reasoning over rate tables and retrieval-heavy policy interpretation rather than deterministic arithmetic.
Ablation analysis confirms that the deterministic modules are individually load-bearing. Removing rolling-year adjustment, exact rate lookup, tenure computation, or premature-withdrawal logic each produces measurable accuracy drops. This is good experimental hygiene and strengthens the causal claims about which components matter.
Implications and Limitations
The architecture-versus-scale finding has broad implications. The AI community has largely assumed that scaling resolves most capability gaps over time. CIFQA presents a clear counterexample: for tasks requiring deterministic correctness, the ceiling on pure LLM performance may be structural rather than a function of parameter count or training data. This aligns with a growing body of literature showing that LLMs exhibit fragility to numerical perturbations and accumulate errors in chained arithmetic, regardless of scale.
The modular design also points toward a general pattern for other calculation-intensive domains: loan amortisation, bond yield estimation, tax computation, and portfolio rebalancing all share the same structural properties that make CIFQA relevant beyond fixed deposits.
That said, several limitations warrant attention. The evaluation is confined to a single financial product type and a relatively small benchmark of 126 queries. Broader validation across instruments and regulatory contexts is necessary before strong generalisability claims can be made. The framework's performance on policy-heavy queries is noticeably weaker, which the authors acknowledge honestly; integrating retrieval and reasoning more tightly with the deterministic execution layer is an open problem. There is also a dependency on accurate parameter extraction and routing at the front end. Errors there propagate downstream with no recovery mechanism described in the paper. In adversarial or ambiguous query conditions, this could be a meaningful vulnerability.
One methodological point deserves scrutiny. Baseline models were evaluated through native conversational interfaces rather than raw API inference, on the grounds that this reflects real-world deployment. This is a reasonable practical choice, but it introduces some opacity about what tool use or internal reasoning the models may have invoked. The comparison is fair in spirit, but readers should note that the experimental conditions are not fully controlled at the model-internals level.
Overall, CIFQA makes a well-argued and empirically grounded case that deterministic execution is not optional for financial AI systems where numerical correctness is a hard requirement. The architectural insight, that LLMs should interpret and plan while deterministic engines compute, is not entirely novel in the tool-augmented reasoning literature, but the systematic application to calculation-intensive financial QA with rigorous evaluation is a genuine contribution. The paper is worth reading for anyone building or evaluating LLM systems in regulated, numerically sensitive domains.