Aloe-Vision: Robust Vision-Language Models for Healthcare
Quick Answer
Aloe-Vision introduces a robust family of healthcare-focused Vision-Language Models (LVLMs) trained on a new dataset, Aloe-Vision-Data, which enhances performance without sacrificing general capabilities.
Quick Take
Aloe-Vision introduces a robust family of healthcare-focused (LVLMs) trained on a new dataset, Aloe-Vision-Data, which enhances performance without sacrificing general capabilities. The models, available in 7B and 72B scales, show significant improvements over baseline models, while CareQA-Vision provides a reliable benchmark for evaluation, highlighting existing vulnerabilities to adversarial inputs.
Key Points
- Aloe-Vision-Data integrates medical and general multimodal data for model fine-tuning.
- The Aloe-Vision LVLMs achieve competitive performance against state-of-the-art models.
- CareQA-Vision benchmark offers low-contamination vision questions for reliable evaluation.
- Current LVLMs are vulnerable to adversarial inputs, posing reliability challenges.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 27500v1 Announce Type: new Abstract: Large (LVLMs) specialized in healthcare are emerging as a promising research direction due to their potential impact in clinical and biomedical applications. However, progress is constrained by the scarcity of high-quality medical multimodal data, concerns about robustness in safety-critical settings, and the narrow and potentially contaminated evaluation benchmarks that limit reliable assessment.
To address these issues, the field requires state-of-the-art solutions to be fully open and reproducible systems in which all components can be inspected, evaluated, and improved. This work introduces Aloe-Vision-Data, a large-scale, quality-filtered mixture which integrates both medical and general domains across multimodal and text-only sources, designed for direct use in model fine-tuning.
Building on this dataset, we train the Aloe-Vision family of medical LVLMs, openly released with full weights, training recipes and data, in two scales (7B and 72B). Through comprehensive benchmarking, we demonstrate that high quality training mixtures produce balanced LVLMs which yield significant gains over the baseline models without compromising general capabilities, achieving competitive performance with respect to state-of-the-art alternatives.
To support reliable evaluation, we introduce CareQA-Vision, a carefully curated vision benchmark derived from MIR and EIR exams, the residency entrance exams for medical and nursing specialists in Spain, offering novel vision questions with low likelihood of contamination. Finally, we show that current LVLMs remain vulnerable to adversarial and misleading inputs, underscoring reliability challenges in clinical contexts.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.
