COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
Quick Answer
COFFEQA introduces a counterfactual-guided few-shot distillation method for LLMs, enhancing QA performance.
Quick Take
Experiments show an 8.77 average EM point and 9.98 average F1 point improvement on TriviaQA using T5 with 32 few-shot pairs, outperforming traditional factual-only distillation methods.
Key Points
- COFFEQA leverages counterfactuals to enhance few-shot distillation for QA tasks.
- New metric, Context Following Difficulty (CFD), validates counterfactuals before distillation.
- Experiments conducted on five in-domain and three out-of-domain QA benchmarks.
- Outperforms factual-only distillation baselines under the same few-shot budget.
- Significant improvements observed in TriviaQA with T5 model.
DeepSignal Analysis
What happened
COFFEQA introduces a method for few-shot distillation in large language models (LLMs) that utilizes counterfactuals to enhance question answering performance. Experiments demonstrate significant improvements over traditional methods, particularly on the TriviaQA benchmark.
Key evidence
- COFFEQA improves average Exact Match (EM) scores by 8.77 points and F1 scores by 9.98 points on TriviaQA using T5 with 32 few-shot pairs.
- The method leverages counterfactuals, defined as minimally modified contexts that prompt a change in the teacher model's response.
- Experiments were conducted across five in-domain and three out-of-domain QA benchmarks, showing COFFEQA's effectiveness compared to factual-only distillation methods.
Why it matters
The development of COFFEQA addresses the limitations of traditional few-shot distillation methods, which often rely on uncertain training samples. By incorporating counterfactuals, the method enhances the learning process, potentially leading to more efficient and effective models in resource-constrained environments.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in Open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by 8.77 average EM points and 9.98 average F1 points under the T5 setting with 32 few-shot pairs.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12837 [cs.CL] |
| (or arXiv:2610.12837v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12837 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kavinda Kehelella [view email]
[v1]
Fri, 9 Oct 2026 00:41:00 UTC (571 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.