Quantum programming sits at an awkward intersection: the hardware demands low-level, timing-precise, pulse-aware control, while most tooling and training data treats quantum programs as high-level gate sequences. This paper addresses a concrete and underappreciated bottleneck: there is no dataset designed to train or evaluate large language models on the hardware-facing features of OpenQASM 3, the language specifically built to expose those low-level controls.
The timing matters. OpenQASM 3 supports mid-circuit measurement and classical feedback for quantum error correction, explicit timing constructs for dynamical decoupling, and pulse-level waveform access for calibration. These are not niche features; they are the primary tools for managing noise in the NISQ era. Yet existing datasets like QCircuitBench and Agent-Q only cover gate-sequence circuit generation, and resources like QASMBench were never intended for LLM training at all. The gap is real, and QASM-Eval is a direct attempt to close it.
What the Dataset Contains
QASM-Eval consists of 4,000 training tasks and 100 expert-verified test tasks. The structure is task-completion style: critical code segments are replaced with natural language prompts, and models must generate the missing OpenQASM 3 implementation. Tasks are organised across four categories:
- Classical logic and control flow: conditionals, loops, runtime feedback
- Timing scheduling: gate alignment, delay insertion, duration constraints
- Pulse control: waveform definition and calibration primitives
- Complex real-world workflows: tasks requiring all of the above simultaneously
Verification is automated using an extended toolchain that checks syntax, simulates quantum states, and validates execution schedules. The verifier was compared against expert annotations on the 100-task test set, achieving 92% agreement and a Cohen's kappa of 0.837. The eight disagreements mostly trace back to ambiguity in natural language task descriptions rather than verifier failures, which is a reasonable result and a known challenge in code-generation benchmarking generally.
Baseline Performance and the Fine-Tuning Effect
The evaluation covers ChatGPT-5.2-Thinking, DeepSeek-V3, Llama-3.3-70B, and Llama-3.1-8B, tested under zero-shot, few-shot (3 exemplars per problem), and LoRA fine-tuning conditions. The headline numbers are striking:
- GPT-5.2 zero-shot achieves 0.54 overall pass@1
- Llama-70B zero-shot reaches only 0.27
- Few-shot prompting pushes GPT-5.2 to 0.78 and Llama-70B to 0.60
- Fine-tuning on QASM-Eval brings Llama-70B to 0.85, exceeding few-shot GPT-5.2
- Fine-tuned Llama-8B reaches 0.52, approaching zero-shot GPT-5.2
The complex task category is particularly revealing. Zero-shot performance across all models sits between 0.00 and 0.08, which reflects the difficulty of simultaneously satisfying OpenQASM 3 syntax constraints and interpreting dense, multi-requirement natural language specifications. Few-shot prompting helps substantially here, especially for larger models: GPT-5.2 jumps from 0.08 to 0.64 on complex tasks with three exemplars. The exemplars appear to do two things at once: expose syntactic patterns and clarify the mapping between task description and implementation intent.
Fine-tuning on prior datasets (QCircuitBench, Agent-Q) provides essentially no benefit on QASM-Eval tasks, which is expected given those datasets do not cover classical logic, timing, or pulse features. This is a useful negative result: it confirms that the performance gains from QASM-Eval fine-tuning are not simply the product of more quantum-adjacent training data, but reflect the specific content of the new dataset.
Error Analysis and Remaining Challenges
The error breakdown across model variants tells a coherent story. Syntax errors are the most common failure mode in base models and the most tractable to fix through fine-tuning. For Llama-70B, syntax errors drop from 311 (base) to 91 (few-shot) to 8 (fine-tuned). This is the expected result: supervised fine-tuning on well-formed code directly targets formatting and structural compliance.
Timing-related semantic errors are more resistant to improvement and show a scale-dependent pattern. Llama-8B shows no meaningful reduction in timeline errors across conditions, while Llama-70B does improve after fine-tuning. This suggests that handling temporal scheduling constraints requires a model capacity threshold that the 8B scale does not reliably reach. As syntax errors are resolved, semantic constraint violations become the dominant failure mode, pointing toward the next generation of training data or verification-guided generation as the likely path forward.
Pulse control tasks are the easiest category across all models, including base models. The authors attribute this to the structured, API-like nature of waveform construction, which resembles patterns more likely to appear in general code pretraining. This is plausible, though it also raises a question about whether pulse-control tasks in the current dataset are genuinely representative of the difficulty practitioners encounter.
Implications and Open Questions
The practical implication is clear: if you want an LLM to assist with hardware-level quantum programming, general-purpose models are insufficient and existing quantum datasets do not help. A relatively modest fine-tuning effort on a targeted dataset can close a large fraction of the gap, even for smaller models.
There are some limitations worth flagging. The test set of 100 tasks is small, which limits the statistical resolution of category-level comparisons. The verifier agreement rate of 92% is strong but not perfect, and the authors acknowledge that prompt ambiguity is a confounding factor. It would also be worth understanding how the dataset generalises to real hardware workflows, since the tasks are synthetically constructed and expert-verified rather than drawn from production quantum programs.
The dataset also does not address quantum algorithm design or reasoning about correctness at the algorithmic level, which is an explicit scope decision. That is reasonable for now, but the longer-term challenge is that hardware-level programming rarely happens in isolation from algorithmic intent. A future dataset that bridges those layers would be considerably harder to construct but potentially more impactful.
For now, QASM-Eval fills a genuine gap and provides a credible foundation for LLM-assisted quantum programming tooling. The dataset, fine-tuned models, and verification framework are all open-sourced at github.com/fuzhenxiao/QASM-Eval, which should make it straightforward for others to build on.