Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance
Quick Answer
This paper shows that The Semantic-Aware Kernel Entropy (SAKE) guidance method enhances text diffusion models by optimizing the balance between fidelity and diversity.
Quick Take
By utilizing order-2 Rényi entropy, it dynamically adjusts sampling distributions, outperforming traditional methods in reasoning-intensive tasks like code and mathematics generation. Empirical results show significant improvements in multi-sample performance compared to temperature scaling and discrete guidance baselines.
Key Points
- Introduces a training-free SAKE guidance method for text diffusion models.
- Utilizes order-2 Rényi entropy to capture semantic interactions and token positions.
- Achieves a superior Pareto frontier between fidelity and diversity.
- Improves multi-sample performance on reasoning-intensive tasks.
- Outperforms temperature scaling and discrete guidance baselines.
Paper Resources
Source Excerpt
Although diffusion models have revolutionized continuous domains like image synthesis through high quality generations and controllable guidance mechanisms, bringing this controllability to the discrete, sequential nature of text remains an open challenge. Meanwhile, current sampling strategies and guidance methods adjust token likelihoods without capturing the broader semantic landscape, leading to a suboptimal balance between fidelity and diversity. In this work, we introduce a novel training-
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.