When an LLM agent needs to select the right API from a catalog of 47,000 tools, the retrieval mechanism matters enormously. The dominant approach, embedding-based retrieval, has well-documented limitations at scale: compact encoders can underspecify the semantics of complex, overlapping APIs, and the retriever and generator train with separate objectives. Parametric tool retrieval, as proposed in ToolGen (Wang et al., 2025), offers an appealing alternative by encoding the entire tool catalog directly into model parameters via virtual tokens. On standard ToolBench benchmarks, it achieves recall above 0.90. The question this paper asks is whether that number means what we think it means.
The answer, according to this work from SAP Labs, is largely no. The paper introduces ToolSense, an open-source diagnostic framework that auto-generates three evaluation benchmarks from any tool catalog, and applies it to ToolBench to expose what the authors call a knowledge-retrieval dissociation: models that look highly capable under standard evaluation conditions fall apart the moment queries deviate from the training distribution. The full paper is available at arXiv:2606.12451, with code and benchmarks released at github.com/SAP/toolsense.
Why Standard Benchmarks Are Misleading
Two design choices in existing ToolBench evaluation combine to mask genuine failure modes. First, the standard query splits (G1/G2/G3) consist of verbose, fully-specified queries generated by GPT-4, which closely match the training distribution. Real users do not write queries this way. Second, constrained decoding via a DisjunctiveTrie means the model only needs to rank paths through a fixed set of valid token sequences rather than freely recall a tool identifier. This is a significant crutch: the model never has to demonstrate that it has actually memorised anything, only that it can score paths the trie already knows about.
The practical stakes are non-trivial. Downstream agentic fine-tuning typically requires free-form token generation without trie support, so a model that depends entirely on the trie at inference time is a silent failure risk in production.
The ToolSense Framework
ToolSense takes a tool catalog as input and generates three benchmarks automatically using an LLM pipeline:
- Realistic Retrieval Benchmark (RRB): queries at three ambiguity tiers designed to reflect how users actually phrase requests, rather than the verbose GPT-4-generated queries in ToolBench training data.
- MCQ probing benchmark: multiple-choice questions testing discriminative factual knowledge about tool capabilities (random baseline 25%).
- QA probing benchmark: yes/no questions testing inferential factual knowledge about tool properties (random baseline 50%).
Alongside the benchmarks, the framework introduces an Internalization Score (IS@k), defined as the ratio of free-form recall to constrained recall. IS close to 1 means the model generates correct tokens equally well without the trie; IS close to 0 means the model is entirely trie-dependent. Crucially, IS is always reported alongside both recall figures, making trie-dependence transparent rather than hidden inside a single aggregate number. The authors also discuss why a difference-based formulation would conflate distinct failure modes, which is a sensible methodological choice.
Benchmark quality was validated by human annotators on 100 samples per benchmark, achieving inter-annotator agreement of κ ≥ 0.805.
Experimental Findings
Five training configurations were evaluated on Gemma3-4B, with cross-architecture validation on Qwen3.5-4B and Gemma3-12B. Configurations varied along three axes: token format (flat single-token vs. hierarchical multi-token), memorisation format (single vs. multi-format with reverse mappings and hard negatives), and training method (full fine-tuning vs. LoRA).
The headline findings are striking:
- Configurations achieving 84-96% constrained recall on standard G1/G2/G3 queries collapse by 50-64 percentage points on RRB queries, falling below embedding-model baselines.
- Despite strong retrieval performance, several models score near-random on MCQ factual probes (around 29-31% against a 25% baseline), suggesting Stage 2 retrieval fine-tuning systematically destroys the tool knowledge built during Stage 1 memorisation.
- Hierarchical tokens exhibit deep trie-dependence (IS: 0.28-0.33) compared to flat tokens (IS: 0.75-0.98), suggesting the sequential generation structure makes free-form recall substantially harder.
- LoRA combined with multi-format Stage 1 memorisation best mitigates knowledge erosion. Gemma3-12B TG-3FM LoRA achieves 76.4% MCQ retention with 48.4% RRB recall. Qwen3.5-4B shows notably better knowledge retention than Gemma3-4B at the same parameter count (62-74% MCQ vs. 29-42%), which the authors attribute to architecture rather than scale.
- A targeted experiment training Stage 2 on RRB-style queries (rather than verbose queries) raises RRB recall from 43.8% to 87.8%, but MCQ accuracy drops further to near-random (26%), confirming the knowledge erosion is caused by the retrieval fine-tuning objective itself, not the query distribution.
The last finding is particularly important. It rules out the simpler explanation that the problem is just a training distribution mismatch, and implicates something more fundamental about how retrieval SFT interacts with the parametric knowledge encoded in Stage 1. This connects to the well-established catastrophic forgetting literature, and the authors' finding that LoRA mitigates the problem is consistent with prior work showing that low-rank updates better preserve pre-trained representations.
Limitations and Broader Implications
The authors are candid about the scope of their empirical study. The analysis covers only the ToolGen two-stage paradigm on ToolBench; whether the same dissociation appears in other parametric retrieval designs is an open question. The benchmarks are LLM-generated and human-validated only on 100 samples per benchmark, which is a reasonable starting point but not definitive at catalog scale. The IS metric can exhibit high variance when constrained recall is low, a limitation the authors acknowledge and handle explicitly. The model experiments cover 4B-12B parameters, leaving open whether very large models show different retention behaviour under sequential fine-tuning.
There is also a potential confound worth noting: open-source base models may have encountered RapidAPI tool documentation during pre-training, which could inflate absolute Stage 1 knowledge scores. The relative forgetting patterns across training stages should be unaffected, but the findings may not transfer directly to private enterprise tool catalogs with no public pre-training exposure.
The broader implication is methodological. The entire class of generative and parametric retrieval systems, including DSI (Tay et al., 2022) and related work, evaluates exclusively with constrained decoding. ToolSense's free-form IS protocol is a straightforward addition that any such system should adopt. High recall under constrained decoding is a necessary but clearly insufficient condition for claiming that a model has genuinely encoded tool semantics. For anyone building LLM agents over large API catalogs, the gap between benchmark performance and realistic query performance documented here is a serious practical concern, and Stage 1 MCQ accuracy appears to be a reliable early predictor of downstream OOD generalisation, which has useful implications for training pipeline design.