EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction
Quick Answer
EURO-5K is a curated dataset for extracting reporting obligations from EU legislation, enabling effective evaluation of BERT-style and LLM models.
Quick Take
EURO-5K is a curated dataset for extracting reporting obligations from EU legislation, enabling effective evaluation of BERT-style and models. Fine-tuned Legal-BERT outperforms generic models in constrained settings, achieving 0.89 F1 score, while demonstrating that legal pretraining enhances early learning efficiency.
Key Points
- EURO-5K includes 136 EU legislative acts with sentence-level reporting obligations.
- Fine-tuned Legal-BERT outperforms generic BERT in constrained adaptation scenarios.
- Models converge around 3K samples, validating dataset sufficiency for extraction tasks.
- Legal pretraining accelerates learning with minimal data, enhancing model performance.
- Cross-dataset evaluations confirm models act as specialized extractors, not generic classifiers.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 02971v1 Announce Type: new Abstract: Extracting reporting obligations from EU legislation is critical for assessing and reducing regulatory reporting burden. However, distinguishing reporting requirements from structurally similar provisions requires specialised legal understanding. Current legal NLP methods lack specialised datasets with clear guidelines and comparative evaluation of extraction paradigms and domain adaptation strategies.
We curate EURO-5K, a corpus of sentence-level reporting obligations and challenging negative examples from 136 EU legislative acts. On this dataset, we train and compare discriminative token-classification models (BERT-style) and generative span-extraction models (), evaluating both full fine-tuning and parameter-efficient QLoRA against baselines (pattern and dependency-based extraction, few-shot prompting). Results show that fully fine-tuned generic and legal BERT models achieve similar performance (0.
89 F1), while fine-tuned LLMs match encoder accuracy for sentence-level extraction. Legal pretraining offers only small gains for generative models. In contrast, it is clearly beneficial when adaptation capacity is constrained, as parameter-efficient tuning of Legal-BERT outperforms its generic counterpart. Learning curve analysis demonstrates that legal pretraining accelerates early learning with minimal data.
All approaches converge around 3K samples with diminishing returns thereafter, validating dataset sufficiency. Cross-dataset evaluation on two external regulatory corpora shows that our models behave as specialised reporting obligation extractors rather than generic regulatory classifiers. We release EURO-5K, trained models, and an interactive demo with explainability visualizations and structured RDF export.
These demonstrate that both paradigms and parameter-efficient training provide practical tools for regulatory compliance automation.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.