SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation
Quick Answer
SynthAVE introduces a scalable benchmark for e-commerce attribute extraction, validating synthetic labels across 12,726 products and 792 attributes in four languages.
Quick Take
SynthAVE introduces a scalable benchmark for e-commerce attribute extraction, validating synthetic labels across 12,726 products and 792 attributes in four languages. Utilizing a multi-LLM arena with 21 judge configurations, it achieves a Cohen's kappa of 0.92, demonstrating high agreement with human experts while enabling cost-effective validation.
Key Points
- SynthAVE spans 12,726 products across 229 types and 792 attributes.
- Validates synthetic labels using a multi-LLM arena framework with 21 configurations.
- Achieves 95.2% agreement with human experts, Cohen's kappa of 0.92.
- Individual judges show substantial inter-model agreement with Fleiss' kappa of 0.76.
- Enables cost-effective validation while maintaining quality parity with human review.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs, deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale human-validated benchmark for attribute value extraction spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where samples are independently evaluated by 21 judge configurations (7 model families $\times$ 3 prompts), with final labels determined via majority voting. The majority vote ensemble agrees with human experts at Cohen's $\kappa = 0.92$ (95.2% agreement), while individual judges show substantial inter-model agreement (Fleiss' $\kappa = 0.76$). This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.07469 [cs.CL] |
| (or arXiv:2607.07469v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07469 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Virginia Negri [view email]
[v1]
Wed, 8 Jul 2026 14:32:28 UTC (5,765 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.