Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
Quick Answer
This study demonstrates that the quality of training data significantly impacts closed-book QA accuracy when documents are internalized into a 4-bit Gemma-4-e4b model via LoRA.
Quick Take
A single curation pass improved accuracy from 57.7% to 85.7% on a 15-document corpus, outperforming traditional retrieval methods like BM25-.
Key Points
- Training data quality is the key factor for closed-book QA accuracy.
- A single curation pass improved accuracy from 57.7% to 85.7%.
- Internalized adapter achieved 84.2% recall, outperforming BM25-RAG.
- Capacity must grow with corpus size for optimal performance.
- Study includes three misdiagnoses as a case study in training.
Paper Resources
Source Excerpt
We study baking documents directly into the weights of a 4-bit Gemma-4-e4b model via LoRA, so a system can answer questions about a corpus closed-book: no retrieval and no context-window budget. Across roughly 100 training runs from single documents to a 99-document corpus, we find that once adapter capacity is adequate, training-data quality is the dominant lever on closed-book accuracy, outweighing LoRA rank, learning rate, and two alternative architectures combined; capacity itself is a hard
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.