Fine-Tuning Vision-Language Models for Understanding Current Damage and Scoring Priority with Quality Guard Agent
Quick Take
This paper presents a methodology for automating bridge damage assessment using fine-tuned LLaVA-1.5-7B Vision-Language Models, achieving a 70.2% reduction in inference time to 10.06 seconds per image. The model, trained on 4,000 paired images and texts, minimizes inter-rater variability and enhances inspection workflows for aging engineers. A two-stage Quality Guard system further ensures the reliability of damage scoring.
Key Points
- Fine-tuned LLaVA-1.5-7B model reduces inference time to 10.06 seconds per image.
- Training on 4,000 paired images and texts minimizes inter-rater variability in assessments.
- Optimal validation loss achieved with 2,000 training samples in just 2.9 hours.
- Semantic similarity peaks at 3,000 samples, indicating quality-curated data is superior.
- Two-stage Quality Guard system filters low-quality outputs before scoring.
Article Content
From source RSS / original summaryarXiv:2605. 27452v1 Announce Type: new Abstract: Bridge inspection in Japan requires mandatory visual assessments every five years, yet qualitative damage ratings (levels a-e) assigned by different engineers exhibit significant inter-rater variability -- a critical barrier to consistent infrastructure management. The aging of skilled engineers further threatens inspection capacity.
This paper presents a methodology for automating bridge damage understanding and repair priority scoring using fine-tuned Vision-Language Models (VLMs). We fine-tune LLaVA-1. 5-7B with QLoRA on up to 4,000 paired bridge damage images and inspection text records, then evaluate on a fixed test set of 800 images. The model outputs natural language descriptions identifying structural members and damage patterns, from which a rule-based scoring engine calculates a five-level repair priority index.
A progressive training study (1k/2k/3k/4k samples) reveals that 2k training samples achieve near-optimal validation loss in only 2. 9 hours of training; beyond 2k, validation loss improves by no more than 0. 2% per doubling of training samples, exhibiting clear diminishing returns. Furthermore, semantic similarity on the held-out test set peaks at 3k (0. 6909) and degrades at 4k (0. 6739), indicating that quality-curated mid-scale data outperforms larger but noisier corpora. Inference optimization combining torch.
compile() and batch processing (batch_size=8) achieves 10. 06 seconds per image -- a 70. 2% reduction over the unoptimized baseline. Our approach contributes to data governance in bridge inspection, reduces inter-rater variability, and provides AI-assisted triage to augment expert engineers in inspection workflows.
Furthermore, we introduce a two-stage Quality Guard using a fine-tuned Swallow-8B SLM to reject low-quality VLM outputs before priority scoring, preventing spurious scores from damaged or unrecognised images.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.
