Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts
Quick Answer
This study presents a multimodal sentiment analysis approach that integrates audio and multilingual text using cross-modal transformers.
Quick Take
This study presents a multimodal sentiment analysis approach that integrates audio and multilingual text using cross-modal transformers. By distilling knowledge from a teacher model to a student model, significant performance improvements were observed in sentiment classification, particularly with automatically generated transcripts and translations. The code for replication is publicly available.
Key Points
- Introduces a cross-modal transformer architecture for audio sentiment analysis.
- Automatically generates multilingual transcripts using ASR and machine translation.
- Demonstrates significant performance boosts in sentiment classification with multimodal data.
- Knowledge distillation enhances an audio-only model without inference overhead.
- Code for the study is publicly released for reproducibility.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models to solve the task, but it remains unclear if such models can take all aspects into account. To this end, we propose a multimodal solution that integrates audio and text information via cross-modal transformers, where text transcripts are automatically generated via an automatic speech recognition (ASR) tool. Moreover, we create multiple text modalities by automatically translating the transcripts into multiple languages via machine translation tools. Audio and multilingual text features are combined via a cascaded architecture comprising cross-modal transformer blocks that integrate modalities one by one. We further distill knowledge from the multimodal model, called teacher, into a unimodal (audio only) model, called student. We conduct experiments on a large-scale dataset, demonstrating that the automatically generated textual information can bring significant performance boosts in multimodal sentiment polarity classification. Our ablation study confirms that both automatic transcripts and automatic translations are helpful. Moreover, we show that the audio-only model can be enhanced via distillation, boosting performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at this https URL.
| Comments: | Accepted at KES 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Sound (cs.SD) |
| Cite as: | arXiv:2607.06611 [cs.CL] |
| (or arXiv:2607.06611v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.06611 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Radu Tudor Ionescu [view email]
[v1]
Tue, 7 Jul 2026 06:48:23 UTC (220 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.