ImmigrationQA: A Source-Grounded Dataset and Small-Model Adaptation for U.S. Immigration Law
Quick Answer
This paper shows that The ImmigrationQA dataset comprises 17,058 question-answer pairs across 13 immigration subdomains, fine-tuned on a Llama 3.2 3B Instruct model using LoRA, achieving a 27% improvement in mean score over the base model.
Quick Take
The system, costing approximately $29 in cloud compute, aids petitioners lacking legal representation but is not a substitute for legal counsel.
Key Points
- Dataset constructed from 11 sources, including USCIS Policy Manual and BIA decisions.
- Fine-tuned model scored 1.08/3.0, outperforming the Llama 3 8B base model at 0.85/3.0.
- Model shows significant improvement in procedural subdomains but struggles with complex legal reasoning.
- All artifacts including dataset and model are publicly available.
- System does not reflect regulatory changes post-corpus crawl date.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 30589v1 Announce Type: new Abstract: U. S. immigration law spans thousands of pages of official policy, federal regulations, and procedural guidance that change frequently and carry high stakes for petitioners who lack legal representation. We describe the construction of ImmigrationQA, a source-grounded question-answering dataset of 17,058 pairs across 13 immigration subdomains, and the fine-tuning of a Llama 3. 2 3B Instruct model on that dataset using parameter-efficient LoRA.
The corpus was assembled from 11 primary and secondary sources -- including the USCIS Policy Manual, 8 CFR, BIA precedent decisions, and community Q&A -- yielding 10,056 validated canonical documents and 18,308 text chunks. Structured QA pairs were generated from these chunks using Claude Sonnet 4. 6 via five mode-specific prompts, with 22 pairs rejected for insufficient source-span overlap.
The fine-tuned model was evaluated against a held-out split of 993 pairs using -as-judge scoring on a 101-example stratified sample. The fine-tuned model scored a mean of 1. 08/3. 0 (16. 8% fully correct; 101-example stratified eval) versus the Llama 3 8B base model at 0. 85/3. 0 (4% fully correct), a relative improvement of 27% in mean score; a zero-shot Claude Sonnet baseline scored 1. 52/3. 0 (25% fully correct).
The fine-tuned model shows concentrated improvement in procedural subdomains (travel documents, adjustment of status, nonimmigrant visas) while remaining weak on complex legal reasoning and time-sensitive statistics. The full pipeline ran for approximately $29 in cloud compute. All artifacts -- dataset, model, code, and prompt templates -- are publicly released. The system is not a substitute for legal counsel and does not reflect regulatory changes after the corpus crawl date.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.