Classical Latin presents a persistent challenge for computational linguistics: the language is morphologically complex, historically significant, and chronically under-resourced relative to modern languages. Annotated corpora exist but remain sparse, and the conventions governing named entities in Latin do not map cleanly onto those of the modern Romance or Germanic languages that dominate contemporary NLP training data. A new paper from the University of Ottawa addresses this problem directly by applying few-shot prompting of commercial LLMs to the EvaLatin 2026 Named Entity Recognition shared task, placing first across all subtasks and evaluation regimes.
The result is practically significant beyond the competition context. It offers evidence that general-purpose LLMs trained predominantly on modern multilingual text can be steered toward competent performance on ancient language tasks without any task-specific fine-tuning, purely through careful prompt construction.
The Task and Why It Is Hard
EvaLatin 2026's NER shared task is divided into two subtasks. The coarse-grained track covers 11 entity classes; the fine-grained track extends this to 28 classes. Both are evaluated under strict and fuzzy regimes. Strict evaluation requires exact token boundary and entity type matching. Fuzzy evaluation relaxes boundary constraints, requiring only a single overlapping token with the gold standard annotation.
Twenty-eight fine-grained entity classes is a demanding label space even for well-resourced languages. For Latin, the difficulty is compounded by the fact that named entity conventions differ substantially from modern languages. Latin common nouns frequently become proper names in English and other derivative languages, a phenomenon that creates systematic misclassification pressure when models trained on modern corpora are applied to Latin text. The word gloria, meaning glory in Latin, is a personal name in English. A model whose named entity intuitions are calibrated on English will tend to tag it as a person, incorrectly.
Methodology
The system uses two commercial LLMs: Gemini 2.5 Pro and Claude Sonnet 4.5. The core prompting strategy is few-shot in-context learning, where each prompt includes annotated examples drawn from the sample data provided by the shared task organisers. Examples were selected randomly from this pool, which the author acknowledges as a limitation worth addressing in future work.
The prompts were constructed to provide the model with the output format, examples of Latin text, and their corresponding named entity annotations. This scaffolding serves two purposes. First, it constrains the output to a parseable structure. Second, and more importantly, it provides domain cues that help activate the model's latent representation of Latin, which is weaker than its representation of high-resource languages but not absent.
A secondary set of experiments, conducted outside the official scoring window, explored fine-tuning LaBERTa, a Latin BERT-based model, on the supplemental dataset for coarse-grained classification. These results are reported for comparison but were not submitted officially.
Results and What They Tell Us
The official results are striking in their margin over competing submissions:
- Coarse-grained strict F1: 0.908 (Claude) and 0.895 (Gemini), against a next-best of 0.714
- Fine-grained strict F1: 0.865 (Gemini) and 0.856 (Claude), against a next-best of 0.384
- Fuzzy evaluation shows similar patterns, with fine-grained F1 reaching 0.887 against a next-best of 0.432
The gap in the fine-grained subtask is particularly large, roughly 0.5 F1 points against the nearest competitor. This suggests that while other approaches struggle to resolve 28-class distinctions, the LLMs retain enough generalised linguistic knowledge to make meaningful fine-grained distinctions even in a low-resource ancient language setting.
A consistent pattern across all conditions is that recall exceeds precision. The models tend to over-classify, tagging non-entities as named entities more often than they miss genuine entities. The author attributes this to the bias introduced by training on modern corpora where Latin-derived common nouns function as proper names. This is a plausible and well-reasoned explanation, and it points toward a systematic rather than random error source, which matters for anyone considering deploying these methods on Latin corpora outside the shared task context.
Few-shot prompting outperforms zero-shot consistently across both models, both subtasks, and both evaluation regimes. This is not surprising given the literature on in-context learning, but it is useful confirmation in a low-resource ancient language setting where the benefit might have been expected to be smaller if the models had weak Latin representations to begin with.
The LaBERTa fine-tuning experiment produced coarse-grained strict F1 of 0.749, which would have placed third overall. This is competitive but clearly below the prompting-based approaches. Given that the fine-tuning was constrained to three entity classes by the available supplemental data, a direct comparison is not entirely fair, but the result does suggest that embedding-based approaches remain viable and potentially more cost-efficient at scale.
Limitations and Open Questions
The paper is candid about its constraints. The test corpus, while genre-diverse, is a small slice of the broader Latin textual tradition. Results on, say, technical or legal Latin texts might differ substantially from the literary and historical texts typically included in shared task data.
The prompt engineering process was informed by the author's prior experience with LLMs across domains, which introduces a form of implicit tuning that is difficult to separate from the method itself. No ablation studies over prompt variations were performed, and no statistical significance testing was conducted. Given the competitive setting and time constraints, this is understandable, but it means the reported margins should be interpreted with some caution as point estimates rather than stable performance bounds.
Example selection within few-shot prompts was random. The author correctly identifies cosine similarity-based retrieval as a promising alternative, following Liu et al. (2022). Selecting examples that are semantically or syntactically similar to the input sentence would likely improve performance, particularly for the fine-grained task where the relevant label space is larger and example relevance matters more.
The reliance on commercial APIs also raises reproducibility concerns. Model versions change, and neither Gemini 2.5 Pro nor Claude Sonnet 4.5 is guaranteed to behave identically across API calls or over time. Open-source alternatives would allow more controlled experimentation, though the author notes that exploring these is a direction for future work.
For the broader ANLP community, the most important implication is methodological. Participating in shared tasks with carefully engineered prompts on commercial LLMs is now a competitive strategy even for ancient languages, and the barrier to entry is low compared to training or fine-tuning specialised models. This democratises access to competitive NER performance but also raises questions about what the shared task format is actually measuring when commercial API access is the primary differentiator.
The paper is available in full at arXiv:2608.04015.