Continuous Audio Thinking for Large Audio Language Models
Quick Answer
This paper shows that The Continuous Audio Thinking (CoAT) framework enhances large audio language models (LALMs) like Qwen2-Audio and Audio Flamingo 3 by preserving acoustic information during response generation, leading to improved performance across audio reasoning and transcription tasks without additional decoding costs.
Quick Take
This method demonstrates significant gains in benchmarks, showcasing the effectiveness of expert distillation in audio processing.
Key Points
- CoAT introduces a continuous latent workspace for organizing acoustic information.
- Performance improvements observed in Qwen2-Audio and Audio Flamingo 3 across multiple benchmarks.
- No additional autoregressive decoding costs compared to baseline models.
- Expert distillation enhances the model's ability to leverage rich acoustic details.
- Results confirm effective propagation of auxiliary supervision to textual responses.
Paper Resources
Source Excerpt
Large audio language models (LALMs) have shown impressive capabilities on diverse audio understanding tasks, ranging from speech transcription to music analysis. However, because LALMs are typically trained to produce text-aligned responses, their hidden states are progressively shaped for text generation rather than for preserving acoustic information. As a result, the diverse acoustic content that audio carries, such as phonetic detail, prosody, sound events, affect, and pitch, is lost along t
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.