Toward Real-Time Sentence-Level Sign Language Translation
Quick Answer
This study presents a real-time sentence-level sign language translation system using a fine-tuned SHuBERT-ByT5 model, achieving a BLEU score of 15.9.
Quick Take
This study presents a real-time sentence-level sign language translation system using a fine-tuned SHuBERT-ByT5 model, achieving a BLEU score of 15.9. The system operates on a Raspberry Pi 4B for camera capture and local display, while intensive processing occurs on a CPU/GPU backend, reducing response latency significantly.
Key Points
- Fine-tuned SHuBERT-ByT5 model achieves BLEU score of 15.9 on test data.
- Real-time system utilizes Raspberry Pi 4B for camera and display functions.
- Response latency reduced from 1.873 to 1.354 seconds, a 27.71% improvement.
- Client-agnostic capture protocol supports various devices like browsers and phones.
- Chunked ingestion and parallelized perception enhance processing efficiency.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Thanh-Hoang Nguyen Doan (The University of Danang - University of Science and Technology)
Abstract:Most sign language understanding systems operate at the level of isolated signs, limiting their usefulness in natural communication. We study sentence-level sign language translation (SLT) with the primary goal of real-time deployment rather than proposing a new translation architecture. We fine-tune a SHuBERT-ByT5 translation stack on a uniformly sampled 9,872-example subset of How2Sign, selected because of compute and storage constraints, using QLoRA while keeping SHuBERT frozen. The model obtains a validation BLEU of 16.7 and, on the test split, BLEU 15.9 and BLEURT 44.7. The main contribution is a hardware-aware streaming system: a Raspberry Pi 4B reference client provides camera capture, local text display, and speech output, while compute-intensive perception and translation run on a CPU/GPU backend. The capture protocol remains client-agnostic, so the same backend can serve a browser, phone, or laptop. Chunked ingestion, bounded queues, parallelized perception, temporal reordering, and a sentence-boundary state machine reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 latency from 2.919 to 2.130 seconds (27.03%) over the complete 9,872-example working subset.
| Comments: | 8 pages, 4 figures, 9 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.09611 [cs.CL] |
| (or arXiv:2607.09611v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09611 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nguyen Doan Thanh Hoang [view email]
[v1]
Fri, 10 Jul 2026 17:11:03 UTC (18 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.
