Most research on LLM-based data-science agents focuses on the model or the workflow strategy. The harness, the system layer responsible for task representation, execution state management, artifact constraints, and evaluation feedback, is typically treated as scaffolding: necessary but not worth studying in its own right. DS-Lighting argues, convincingly, that this is a mistake. The paper makes the case that end-to-end agent performance is as much a function of harness design as it is of model capability or workflow strategy, and it provides a toolkit to test that claim empirically.
This matters because the field currently has no reliable way to attribute performance differences across data-science agents. When DS-Agent outperforms AIDE on a given task, is that because of its workflow, its prompt design, or the implicit assumptions baked into its execution environment? Without a shared harness, the answer is unknowable. DS-Lighting is an attempt to fix that.
Key Contributions
The paper makes three substantive contributions:
- A formal decomposition of the agent harness into four reusable layers: data (converting raw tasks into structured contracts), workflow (representing agents as executable operator programs), execution (sandboxed stateful runtime control), and evaluation (step-level and task-level artifact validation).
- An operator-program abstraction that can represent both fixed-pipeline agents (DS-Agent, Data Interpreter, AutoKaggle) and adaptive search agents (AIDE, AutoMind, ReAct) under a common interface, making apples-to-apples comparison tractable.
- A unified evaluation suite that converts DABench, DACode, MosciBench, and ScienceAgentBench into a shared MLE-Bench-style task format, with a single-line invocation per benchmark and a controlled sandbox runtime.
The evaluation layer is worth dwelling on. DS-Lighting distinguishes between step-level evaluation, which checks intermediate artifacts during execution (file structure, schema validity, missing outputs), and task-level evaluation, which scores the completed workflow against official benchmark metrics. This two-level design allows agents to repair failures before final scoring, which is closer to how a competent human analyst would work and removes a class of avoidable system-level errors from the results.
Methodology and Experimental Design
Seven agents are reproduced within DS-Lighting: AutoKaggle, Data Interpreter, DS-Agent, DeepAnalyze, AIDE, AutoMind, and ReAct. All run inside a shared sandbox with the same dependencies, step budget, and evaluation protocol. The primary backbone is DeepSeek-V3.1-Terminus, with a secondary sweep across GPT-5.5, Claude-Opus-4.7, GPT-5.4-mini, Gemini-3.1-Pro, and Qwen3.5-Plus to disentangle model effects from harness effects.
For the harness comparison (RQ2), DS-Lighting is evaluated against VanillaHarness, LangChain, AutoGen, and OpenHands. All receive identical task contracts, public input files, model access, and step budgets. DS-Lighting's additional checks are restricted to public output requirements: artifact existence, filenames, directory structure, and schema consistency. No hidden labels or intermediate metric scores are exposed. This is a clean experimental design that genuinely isolates harness-level behaviour.
The ablation study (RQ3) tests the contribution of individual harness components. The failure-mode analysis uses an LLM-as-judge protocol to categorise errors across grounding mistakes, execution-control failures, evaluation-alignment issues, and intrinsic model limitations, which is a sensible taxonomy for separating fixable system failures from genuine capability gaps.
Results and What They Tell Us
The headline numbers from Table 1 show that workflow style interacts with task type in predictable ways. Fixed-pipeline agents (Data Interpreter, DS-Agent) perform reliably on standardised analyses: Data Interpreter reaches 0.9155 on Correlation Analysis and 0.8864 on Summary Statistics under DeepSeek-V3.1-Terminus. Adaptive agents (AIDE, ReAct) show higher variance, occasionally topping individual task categories but less consistently across the board. This is not surprising, but having it demonstrated under a controlled harness is more convincing than prior comparisons where harness differences confounded results.
The harness comparison in Table 2 is the more interesting result. DS-Lighting matches or exceeds all baselines across all four benchmarks. On DABench it ties with VanillaHarness (0.8521), which suggests that for relatively standardised tasks a lightweight harness is sufficient. The gaps open up on harder benchmarks: DACode (0.3709 vs. 0.3073 for VanillaHarness), MosciBench (0.6477 vs. 0.5000 for LangChain), and ScienceAgentBench (0.1961 vs. 0.1863). The pattern is consistent with the paper's central claim: harness design matters most when tasks are heterogeneous and require richer data access, code execution, and artifact validation.
Perhaps the most striking finding comes from the model sensitivity analysis. DS-Lighting paired with Qwen outperforms LangChain paired with GPT-5.5. If that result holds up to scrutiny, it is a strong argument that infrastructure quality can compensate for model scale, at least within the range of models tested. The failure-mode breakdown supports the same conclusion: DS-Lighting primarily reduces grounding errors, runtime control failures, and evaluator-alignment issues rather than masking reasoning errors. The remaining failures after DS-Lighting's interventions concentrate on intrinsic model limitations, which is the right place for them to be.
Limitations and Open Questions
The authors are straightforward about the scope of their evaluation. DS-Lighting is tested on public benchmarks; real industrial workflows involve private databases, streaming data, human review cycles, and domain governance constraints that none of the benchmarks capture. The sandboxed runtime also introduces its own fragility: results can vary with dependency versions, API availability, and resource limits, which the paper acknowledges but does not fully resolve.
A few questions the paper does not fully address are worth flagging. First, the operator-program abstraction is described at a high level, but the expressiveness limits of that abstraction are not characterised. Can it represent multi-agent collaboration patterns or does it require extension? Second, the LLM-as-judge failure categorisation, while useful, introduces its own reliability concerns; inter-rater agreement or calibration against human annotation would strengthen those claims. Third, the benchmarks used (DABench, DACode, MosciBench, ScienceAgentBench) are all public and relatively well-studied; it would be useful to know how DS-Lighting behaves on tasks where the correct output is less clearly defined.
The broader implication is that the field needs more papers like this one. As data-science agent research matures, the inability to attribute performance differences to specific design choices is a genuine scientific problem. DS-Lighting does not solve it completely, but it provides a concrete framework and a reproducible codebase (available on GitHub) for making progress. Treating the harness as a first-class research object rather than an implementation detail is the right framing, and the experimental evidence here supports it.