Benchmarking Large Language Models for Safety Data Extraction
Quick Answer
This study benchmarks Gemini 1.5 Pro, GPT-4o, Claude 3.7 Sonnet, and Llama 3.1-70B for Safety Data Sheet extraction, revealing Gemini 1.5 Pro achieved the highest accuracy at 84%.
Quick Take
Despite strong performance, no model met the 90% accuracy threshold necessary for reliable industrial use, indicating a need for further fine-tuning and domain adaptation.
Key Points
- Gemini 1.5 Pro outperformed others with 84% accuracy in SDS extraction.
- Text-based extraction methods consistently surpassed multimodal approaches.
- No model achieved the 90% accuracy threshold for reliable industrial deployment.
- Future research should focus on domain-specific training and model calibration.
- Human-in-the-Loop verification is essential for safety-critical applications.
Paper Resources
Source Excerpt
arXiv:2606. 11204v1 Announce Type: new Abstract: Accurate extraction of structured information from Safety Data Sheets (SDS) remains challenging in industrial safety due to heterogeneous document formats and the limitations of traditional rule-based methods. This study benchmarks state-of-the-art (LLMs) for automated SDS data extraction, comparing text-based and multimodal processing pipelines. We systematically evaluate four models: Gemini 1. 5 Pro, GPT-4o, Claude 3. 7 Sonnet, and Llama 3.
1-70B, across three prompting strategies: zero-shot, few-shot, and chain-of-thought. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.