Toward Real-Time Sentence-Level Sign Language Translation
Quick Answer
This study presents a real-time sentence-level sign language translation system using a fine-tuned SHuBERT-ByT5 model, achieving a BLEU score of 15.9.
Quick Take
The system operates on a Raspberry Pi 4B for camera capture and local display, while intensive processing occurs on a CPU/GPU backend, reducing response latency significantly.
Key Points
- Fine-tuned SHuBERT-ByT5 model achieves BLEU score of 15.9 on test data.
- Real-time system utilizes Raspberry Pi 4B for camera and display functions.
- Response latency reduced from 1.873 to 1.354 seconds, a 27.71% improvement.
- Client-agnostic capture protocol supports various devices like browsers and phones.
- Chunked ingestion and parallelized perception enhance processing efficiency.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Thanh-Hoang Nguyen Doan (The University of Danang - University of Science and Technology)
Abstract:Most sign language understanding systems operate at the level of isolated signs, limiting their usefulness in natural communication. We study sentence-level sign language translation (SLT) with the primary goal of real-time deployment rather than proposing a new translation architecture. We fine-tune a SHuBERT-ByT5 translation stack on a uniformly sampled 9,872-example subset of How2Sign, selected because of compute and storage constraints, using QLoRA while keeping SHuBERT frozen. The model obtains a validation BLEU of 16.7 and, on the test split, BLEU 15.9 and BLEURT 44.7. The main contribution is a hardware-aware streaming system: a Raspberry Pi 4B reference client provides camera capture, local text display, and speech output, while compute-intensive perception and translation run on a CPU/GPU backend. The capture protocol remains client-agnostic, so the same backend can serve a browser, phone, or laptop. Chunked ingestion, bounded queues, parallelized perception, temporal reordering, and a sentence-boundary state machine reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 latency from 2.919 to 2.130 seconds (27.03%) over the complete 9,872-example working subset.
| Comments: | 8 pages, 4 figures, 9 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.09611 [cs.CL] |
| (or arXiv:2607.09611v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09611 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nguyen Doan Thanh Hoang [view email]
[v1]
Fri, 10 Jul 2026 17:11:03 UTC (18 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.