Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Quick Answer
The proposed Token-Level Visual-Sensitivity Steering (TLVS) effectively mitigates hallucinations in large vision language models (LVLMs) by refining token-level steering vectors and adaptively modulating steering strength during decoding.
Quick Take
The proposed Token-Level Visual-Sensitivity Steering (TLVS) effectively mitigates hallucinations in large vision language models (LVLMs) by refining token-level steering vectors and adaptively modulating steering strength during decoding. Evaluated on benchmarks like POPE and AMBER, TLVS shows consistent improvements over existing methods, enhancing performance while requiring minimal calibration training.
Key Points
- TLVS refines token-level steering vectors for targeted hallucination mitigation.
- The method adapts steering strength at each decoding step for better control.
- Evaluated on benchmarks like POPE, AMBER, and HallusionBench, showing consistent improvements.
- Requires minimal training for calibration, making it lightweight and efficient.
- Addresses issues of over-perturbation and low signal-to-noise ratios in existing methods.
Article Content
From source RSS / original summaryarXiv:2606. 07647v1 Announce Type: new Abstract: Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time.
However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability.
To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models.
It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.