GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus
Quick Answer
GLAN-QnA-KR is a newly released synthetic Korean instruction-QA corpus with 303,581 rows, generated using Microsoft's Phi-3.5-MoE-instruct.
Quick Take
It features a unique seedless taxonomy-driven approach, ensuring minimal duplication and contamination, making it the largest of its kind on the Hugging Face Hub.
Key Points
- Corpus contains 303,581 rows with a flat taxonomy of 1,084 disciplines.
- Only one exact duplicate question found across the entire dataset.
- Contamination audit shows maximum Jaccard index of 0.163 with no high duplicates.
- Median length of questions is 313 characters; answers average 1,098 characters.
- Released under OpenRAIL license, suitable for downstream citation.
DeepSignal Analysis
What happened
The GLAN-QnA-KR corpus has been released, consisting of 303,581 rows of synthetic Korean instruction-QA data. It was generated using Microsoft's Phi-3.5-MoE-instruct model and employs a seedless taxonomy-driven approach to minimize duplication.
Key evidence
- The corpus includes 1,084 English-labelled disciplines paired with Korean question/answer text, with a median of 313 question characters and 1,098 answer characters per record.
- A contamination audit revealed a maximum character-trigram Jaccard of 0.163, indicating minimal overlap with existing datasets, and no test items exceeded a Jaccard of 0.7.
- At the time of release, GLAN-QnA-KR is the largest synthetic Korean instruction corpus verifiable on the Hugging Face Hub, built under a seedless taxonomy-driven protocol.
Why it matters
This release is significant as it provides a large-scale, openly redistributable resource for Korean instruction-based question-answering tasks. The unique generation method aims to reduce contamination from existing datasets, which is crucial for training robust AI models. The corpus's size and structure could enhance research and development in natural language processing for the Korean language.
Paper Resources
Source Excerpt
We release GLAN-QnA-KR, a 303,581-row openly redistributable Korean instruction-QA corpus produced via the seedless taxonomy-driven GLAN synthesis pipeline with Microsoft's Phi-3. 5-MoE-instruct as the producer model (generation: 2024-12; release: 2024-12; licence: OpenRAIL). The corpus spans a flat taxonomy of 1,084 English-labelled disciplines paired with Korean question/answer text, a 100-900 difficulty scale, and a median of 313 question characters and 1,098 answer characters per record. Two
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.