Most LLM-based systems, even well-engineered ones with tool access, operate in a fundamentally reactive mode: they retrieve information, generate code, or call isolated functions. What they do not naturally do is experiment. Controlled experimentation, the practice of varying one factor while holding others fixed and comparing outcomes against a reference, is the basic epistemic engine of science and engineering. This paper asks whether LLM agents can be structured to operate that way, and demonstrates a working system in the context of pharmaceutical process design. The full paper is available at arXiv:2608.23622.
The problem being addressed is more subtle than it first appears. LLMs can produce plausible-sounding process recommendations from training knowledge alone, but plausibility is not the same as correctness, and in industrial engineering contexts the difference matters. A recommendation to increase cooling rate in a crystallization process might be directionally sensible but quantitatively wrong, or correct under one set of baseline conditions and counterproductive under another. Without a mechanism for testing hypotheses against a model of the actual system, the agent is essentially guessing with confidence. This paper's central contribution is an architecture that closes that loop.
Architecture and Methodology
The proposed system is a six-agent pipeline, five LLM-driven and one deterministic, each assigned a distinct functional role. The division of labour is worth examining carefully because it reflects a deliberate design philosophy: separating abstract reasoning from concrete execution, and interpretation from synthesis.
- Requirement Analyzer: Parses the user query and produces a structured task description, including explicit requirements that downstream agents inherit.
- Planner: Generates an abstract, step-by-step plan referencing available simulation functions by description, without committing to execution details. This mirrors the separation of planning from execution seen in frameworks like Plan-And-Act.
- Interactive Operator: Concretises the plan, generates Python function calls with fully specified parameters, and manages a dialogue loop with the Executor. Critically, it can recover from failed calls by revising and retrying, up to a predefined limit.
- Executor: A deterministic software component that runs simulation code in a controlled interpreter and returns both textual and graphical outputs. It does no language reasoning.
- Interpreter: A vision-capable LLM that reads simulation plots and converts them into textual insights, feeding back into the reasoning chain.
- Reporter: Aggregates all intermediate outputs into a user-facing recommendation.
The simulation model itself is a proprietary industrial asset built on first-principles and semi-empirical equations, capable of reproducing mass transfer, phase transitions, and time-evolving material distributions in crystallization. It is exposed to agents as callable functions with standardised specifications, an approach consistent with the EASYTOOL line of work on clearer tool descriptions.
The overall flow is: query to structured task, task to abstract plan, plan to parameterised simulation experiments, simulation outputs to interpreted evidence, evidence to final recommendation. Each step produces an explicit artefact that the next agent consumes. This makes the reasoning chain traceable, which matters both for debugging and for user trust.
Evaluation and Results
The authors evaluate four system variants across five crystallization optimisation tasks, all agent-based variants powered by GPT-4o: the full system, a version with simulation disabled, a version with the Requirement Analyzer removed, and a vanilla LLM baseline. The evaluation combines lexical specificity metrics, user-rated correctness and helpfulness scores from domain specialists, and simulation call precision and recall.
Key findings:
- The full system produces the fewest vague expressions (13.7 per 1,000 words versus 57.1 for the LLM-only baseline) and the lowest uncertainty score under the LUCI metric (0.13 versus 0.36).
- Correctness and helpfulness scores from domain specialists averaged 4.1 and 4.2 respectively for the full system; both ablated variants without simulation scored below 3.0.
- Simulation call precision reached 94% in the full system, with 76% of generated hypotheses confirmed by simulation outcomes, completing the hypothesis-experiment-validation loop.
- Removing the Requirement Analyzer increased simulation recall to 69% but dropped precision to 65%, suggesting the system over-calls simulation functions without the structured task representation to guide it.
The precision-recall trade-off in the no-requirement condition is telling. More simulation calls does not mean better reasoning; targeted calls grounded in explicit task requirements produce more useful evidence. This aligns with findings from the ReAct and intermediate-reasoning-step literature more broadly.
Limitations and Open Questions
Several limitations deserve attention. The simulation model is proprietary and the evaluation covers only five tasks in a single domain, so generalisation claims are necessarily tentative. The authors acknowledge this and frame the work as a demonstration of a generalizable approach rather than a domain-general system, which is fair, but external validation across different simulation environments would significantly strengthen the case.
The concept of "approximation fidelity" introduced in the discussion is useful but underspecified. The authors argue that system quality is jointly governed by the physical fidelity of the simulator and the hypothesis correctness of the LLM. This is intuitively right, but the paper does not provide a formal treatment or empirical decomposition of how errors from each source propagate through the pipeline. Understanding failure modes, particularly cases where a high-fidelity simulator confirms a poorly formed hypothesis or where a good hypothesis is tested against an inadequately calibrated model, would be valuable for practitioners considering deployment.
The retry mechanism for failed simulation calls is pragmatically useful but raises questions about consistency. If the Operator revises function calls iteratively, the final executed experiment may differ non-trivially from the one originally planned. Whether this introduces systematic bias in which hypotheses get tested is not analysed.
There is also the question of how the system handles genuinely open-ended optimisation problems where the hypothesis space is large and the simulation is expensive. The current framework appears well-suited to targeted, query-driven tasks, but scaling to broader design-space exploration would require integration with something like Bayesian optimisation or active learning, neither of which is addressed here.
Implications
Despite these open questions, the paper makes a genuinely useful contribution. The core insight, that LLM agents should be structured to intervene and compare rather than simply retrieve and generate, is both principled and practically consequential. Coupling hypothesis generation with mechanistic simulation provides a form of grounding that language-only reasoning cannot replicate, and the architecture is modular enough that the simulation component could be swapped out for other high-fidelity models in different domains.
For researchers working on scientific AI, the framing of the system as an implementation of the scientific method (hypothesize, intervene, observe, report) is a useful conceptual anchor. It suggests that the right question for future LLM agent design is not only "how capable is the language model" but "how is it connected to structured sources of experimental evidence." That reframing has implications well beyond pharmaceutical engineering.
Read the full paper at arXiv:2608.23622.