Retrieval-Based Multi-Label Legal Annotation: Extensible, Data-Efficient and Hallucination-Free
Quick Answer
The study presents a retrieval-based approach for multi-label legal annotation, achieving a Macro-F1 score of 49.12 on the Eurlex dataset with Qwen-8B, significantly improving efficiency by reducing compute costs by 20-30 times compared to fine-tuning methods.
Quick Take
The study presents a retrieval-based approach for multi-label legal annotation, achieving a Macro-F1 score of 49.12 on the Eurlex dataset with Qwen-8B, significantly improving efficiency by reducing compute costs by 20-30 times compared to fine-tuning methods. This method eliminates hallucination issues seen in generative models like GPT-5.2, making it a robust alternative for dynamic legal label spaces.
Key Points
- Retrieval model outperforms GPT-5.2, achieving 49.12 Macro-F1 on Eurlex.
- Reduces compute requirements by 20-30 times compared to fine-tuning.
- Only 100 training samples nearly double Micro-F1 over Legal-BERT on ECtHR-A.
- Generative models like GPT-5.2 hallucinate labels in 0.12-0.9% of cases.
- Retrieval-based annotation strictly adheres to defined label sets, avoiding hallucination.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Multi-label legal annotation requires assigning multiple labels from large, evolving taxonomies to long, fact-intensive documents, often under limited supervision. Parametric encoders typically require task-specific training and retraining when the label set changes, while prompting generative large language models becomes costly and degrades as the label space grows. We cast legal annotation as retrieval: we embed documents and label descriptions with a frozen retrieval model and predict labels via k-nearest neighbors in the embedding space, enabling updates by re-embedding and re-indexing rather than gradient-based backpropagation. Across three legal datasets (ECtHR-A, ECtHR-B, and Eurlex with 100 labels), retrieval achieves competitive accuracy and strong data efficiency; on Eurlex, Qwen-8B retrieval improves Macro-F1 from 40.41 (GPT-5.2, zero-shot) to 49.12 while reducing estimated compute by 20-30 times compared to fine-tuning. With only (N=100) training samples, retrieval nearly doubles Micro-F1 over hierarchical Legal-BERT on ECtHR-A (48.29 vs. 27.87). We also quantify a reliability failure mode of generative inference: GPT-5.2 hallucinates labels outside the provided taxonomy in 0.12-0.9% of test samples under deterministic decoding. In contrast, retrieval strictly respects defined label sets, eliminating hallucination by design. These results suggest retrieval-model-based annotators are a practical, deployable alternative for high-cardinality and rapidly changing legal label spaces.
| Comments: | 10 pages, 3 figures |
| Subjects: | Computation and Language (cs.CL) |
| MSC classes: | 68T50 |
| ACM classes: | I.2.7; I.2.4 |
| Cite as: | arXiv:2605.16767 [cs.CL] |
| (or arXiv:2605.16767v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16767 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Li Zhang [view email]
[v1]
Sat, 16 May 2026 02:40:01 UTC (411 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.