EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction
Quick Take
EURO-5K is a curated dataset for extracting reporting obligations from EU legislation, enabling effective evaluation of BERT-style and LLM models. Fine-tuned Legal-BERT outperforms generic models in constrained settings, achieving 0.89 F1 score, while demonstrating that legal pretraining enhances early learning efficiency.
Key Points
- EURO-5K includes 136 EU legislative acts with sentence-level reporting obligations.
- Fine-tuned Legal-BERT outperforms generic BERT in constrained adaptation scenarios.
- Models converge around 3K samples, validating dataset sufficiency for extraction tasks.
- Legal pretraining accelerates learning with minimal data, enhancing model performance.
- Cross-dataset evaluations confirm models act as specialized extractors, not generic classifiers.
Article Content
From source RSS / original summaryarXiv:2606. 02971v1 Announce Type: new Abstract: Extracting reporting obligations from EU legislation is critical for assessing and reducing regulatory reporting burden. However, distinguishing reporting requirements from structurally similar provisions requires specialised legal understanding. Current legal NLP methods lack specialised datasets with clear guidelines and comparative evaluation of extraction paradigms and domain adaptation strategies.
We curate EURO-5K, a corpus of sentence-level reporting obligations and challenging negative examples from 136 EU legislative acts. On this dataset, we train and compare discriminative token-classification models (BERT-style) and generative span-extraction models (LLMs), evaluating both full fine-tuning and parameter-efficient QLoRA against baselines (pattern and dependency-based extraction, few-shot prompting). Results show that fully fine-tuned generic and legal BERT models achieve similar performance (0.
89 F1), while fine-tuned LLMs match encoder accuracy for sentence-level extraction. Legal pretraining offers only small gains for generative models. In contrast, it is clearly beneficial when adaptation capacity is constrained, as parameter-efficient tuning of Legal-BERT outperforms its generic counterpart. Learning curve analysis demonstrates that legal pretraining accelerates early learning with minimal data.
All approaches converge around 3K samples with diminishing returns thereafter, validating dataset sufficiency. Cross-dataset evaluation on two external regulatory corpora shows that our models behave as specialised reporting obligation extractors rather than generic regulatory classifiers. We release EURO-5K, trained models, and an interactive demo with explainability visualizations and structured RDF export.
These demonstrate that both paradigms and parameter-efficient training provide practical tools for regulatory compliance automation.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.