Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars
Quick Answer
The NEST-V1 framework enables emotion-conditioned Nepali Sign Language avatars from spoken input, achieving 81.1% ASR and 79.21% emotion recognition accuracy with a lightweight model of 22.1M parameters.
Quick Take
The NEST-V1 framework enables emotion-conditioned Nepali Sign Language avatars from spoken input, achieving 81.1% ASR and 79.21% emotion recognition accuracy with a lightweight model of 22.1M parameters. This pilot study lays the groundwork for real-time, expressive sign language communication for the hearing-impaired in low-resource settings.
Key Points
- NEST-V1 demonstrates feasibility for emotion-aware sign language translation.
- Achieved 81.1% ASR accuracy and 79.21% emotion recognition accuracy.
- Utilizes a shared acoustic encoder for efficiency with only 22.1M parameters.
- Focuses on four common Nepali words across three emotional states.
- Establishes a scalable framework for future sign language vocabulary expansion.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 26107v1 Announce Type: new Abstract: Sign language communication systems, that integrate emotional expression remain underexplored, particularly for low-resource languages. This pilot study presents NEST-V1 (Nepali Emotion and Speech Transformer - Version 1), a proof-of-concept multimodal framework that demonstrates the feasibility of generating emotion-conditioned Nepali Sign Language avatars from spoken input.
As a preliminary investigation, we focus on four common Nepali words ("thank you", "hello", "house", "me") across three emotional states (happy, neutral, sad) to validate our core technical approach. Our lightweight architecture employs a shared acoustic encoder for simultaneous Automatic Speech Recognition and emotion classification, achieving 81. 1% ASR accuracy and 79. 21% emotion recognition accuracy on a dataset of 600 labeled audio samples from 50 speakers.
The system demonstrates 37% parameter efficiency compared to separate model architectures while maintaining a lightweight footprint with only 22. 1M parameters suitable for edge deployment. This pilot work establishes the technical foundation for emotion-aware sign language translation in low-resource settings and provides a scalable framework for future expansion to larger vocabularies and more diverse emotional expressions.
Our preliminary results indicate the viability of real-time, emotionally expressive sign language communication systems for the hearing-impaired community, with clear pathways for enhancement in subsequent development phases.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.