TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation
Quick Answer
The TAKE framework enables text dataset distillation, reducing corpora to 0.1% size while maintaining task fidelity.
Quick Take
The TAKE framework enables text dataset distillation, reducing corpora to 0.1% size while maintaining task fidelity. By using influence functions, it scores samples based on their contribution to downstream tasks, achieving high data efficiency in text classification and natural language inference without sacrificing accuracy.
Key Points
- TAKE reduces dataset size to 0.1% while preserving downstream task fidelity.
- Utilizes influence functions to quantify sample contributions effectively.
- Achieves high accuracy in text classification and natural language inference.
- Theoretically grounded approach with implications for coreset construction.
- Source code available for further research and application.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large-scale text corpora have become a quiet bottleneck in modern NLP, not just in storage, but in the accumulated cost of training, fine-tuning, and continual learning. We propose a text dataset distillation framework that reduces corpora to as little as 0.1% of their original size while preserving downstream task fidelity. We approach distillation through the lens of influence functions, which quantify each sample's contribution to the downstream objective, a natural and principled basis for selection. We introduce Trajectory-Aware Knowledge Estimation (TAKE), which convolves the knowledge-based influence along the training trajectory into a single per-sample knowledge score, capturing informative samples. These scores serve as sample weights within a discrete Optimal Transport objective, guiding prototype selection from a synthetically generated candidate pool. We evaluate TAKE on downstream accuracy across text classification and natural language inference tasks at extreme compression (0.1% or 20 samples/class), showing that data efficiency is achievable without sacrificing task fidelity. The approach is theoretically grounded, with broader implications for coreset construction and data-centric AI. We release our source code at this https URL.
| Comments: | To be published in ECML-PKDD 2026 |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.11898 [cs.CL] |
| (or arXiv:2607.11898v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.11898 arXiv-issued DOI via DataCite |
Submission history
From: Tri-Nhan Vo [view email]
[v1]
Sat, 13 Jun 2026 05:57:08 UTC (31 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.