Low Resource Multimodal Translation of Nepali Spoken Words into Emotion-Conditioned Sign Language Avatars
Quick Answer
The NEST-V1 framework enables emotion-conditioned Nepali Sign Language avatars from spoken input, achieving 81.1% ASR and 79.21% emotion recognition accuracy with a lightweight model of 22.1M parameters.
Quick Take
This pilot study lays the groundwork for real-time, expressive sign language communication for the hearing-impaired in low-resource settings.
Key Points
- NEST-V1 demonstrates feasibility for emotion-aware sign language translation.
- Achieved 81.1% ASR accuracy and 79.21% emotion recognition accuracy.
- Utilizes a shared acoustic encoder for efficiency with only 22.1M parameters.
- Focuses on four common Nepali words across three emotional states.
- Establishes a scalable framework for future sign language vocabulary expansion.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Sign language communication systems, that integrate emotional expression remain underexplored, particularly for low-resource languages. This pilot study presents NEST-V1 (Nepali Emotion and Speech Transformer - Version 1), a proof-of-concept multimodal framework that demonstrates the feasibility of generating emotion-conditioned Nepali Sign Language avatars from spoken input. As a preliminary investigation, we focus on four common Nepali words ("thank you", "hello", "house", "me") across three emotional states (happy, neutral, sad) to validate our core technical approach. Our lightweight architecture employs a shared acoustic encoder for simultaneous Automatic Speech Recognition and emotion classification, achieving 81.1% ASR accuracy and 79.21% emotion recognition accuracy on a dataset of 600 labeled audio samples from 50 speakers. The system demonstrates 37% parameter efficiency compared to separate model architectures while maintaining a lightweight footprint with only 22.1M parameters suitable for edge deployment. This pilot work establishes the technical foundation for emotion-aware sign language translation in low-resource settings and provides a scalable framework for future expansion to larger vocabularies and more diverse emotional expressions. Our preliminary results indicate the viability of real-time, emotionally expressive sign language communication systems for the hearing-impaired community, with clear pathways for enhancement in subsequent development phases.
| Comments: | 15 pages, 5 figures, 9 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.10; I.2.6; K.4.2 |
| Cite as: | arXiv:2606.26107 [cs.CL] |
| (or arXiv:2606.26107v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.26107 arXiv-issued DOI via DataCite |
Submission history
From: Jatin Bhusal [view email]
[v1]
Mon, 4 May 2026 15:17:49 UTC (788 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.