Adversarially Robust Abductive Fusion of Pre-trained Transformer-based Perception Models
Quick Answer
This study presents a novel method for improving the accuracy of pre-trained ViT-based perception models under distributional shifts by utilizing a learned metacognitive layer that detects errors without domain knowledge.
Quick Take
The approach matches the performance of traditional majority voting methods while significantly outperforming them under coordinated label-flipping attacks, achieving a 22% relative gain in F1 score at a 90% flip rate.
Key Points
- Introduces a metacognitive layer for error detection in perception models.
- Achieves F1 score parity with domain-knowledge rules within 0.002.
- Outperforms majority voting methods under coordinated label-flipping attacks.
- Maintains performance with a 0.42 F1 score at 90% flip rate.
- Utilizes vector-space geometry for consistency-based abduction problem solving.
DeepSignal Analysis
What happened
A new method for enhancing the performance of ViT-based perception models under distributional shifts has been introduced. This method employs a learned metacognitive layer that detects errors without relying on domain knowledge, achieving competitive results against traditional majority voting methods, particularly under coordinated label-flipping attacks.
Key evidence
- The proposed metacognitive layer utilizes vector-space geometry to create Label Vector Pools (LVP) from training embeddings, allowing for error detection without domain knowledge.
- In tests, the new method matched the performance of the best majority-vote variant on clean data, achieving a difference of only $0.005$ in F1 score.
- Under a coordinated label-flipping attack with a 90% flip rate, the method averaged $0.42$ F1, outperforming the majority-vote baseline, which averaged $0.35$, resulting in a 22% relative gain.
Why it matters
This research addresses a significant challenge in deploying AI models in unfamiliar environments, where traditional methods often fail. By eliminating the need for domain-specific knowledge, the approach broadens the applicability of ViT-based models, making them more robust to various real-world scenarios. The ability to maintain performance under adversarial conditions is crucial for applications in critical fields such as autonomous driving and surveillance.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Deploying pre-trained perception models in novel environments degrades their accuracy under distributional shift, and assembling them alone does not recover it: combiners such as majority voting trade recall for precision and are brittle to coordinated failures. Prior metacognitive methods learn logical rules that flag a model's errors, but rely on hand-authored domain-knowledge cues (object-size priors, segmentation masks) that do not transfer to genuinely novel scenes. We show that this metacognitive layer can be learned without any domain knowledge by exploiting vector-space geometry: per-model Label Vector Pools (LVP), built from each model's own training embeddings, yield error-detection rules from the geometry of detections relative to training-determined prototypes, reaching parity with domain-knowledge rules to within $0.002$ every F1 on test set. Because the approach remains neurosymbolic, these geometric rules share a single logical framework and can still be complemented by domain knowledge when available. We frame the fusion of multiple imperfect ViT-based detectors as a consistency-based abduction problem solved at test time by an exact Integer Program (IP) and a polynomial-time heuristic. On an aerial-imagery benchmark of 15 weather-shifted test sets and six ViT detectors, our domain-knowledge-free layer matches the strongest majority-vote variant on clean data (within $0.005$ F1) and, unlike every majority-vote baseline, retains its performance under a coordinated label-flipping attack: at a $90\%$ flip rate it averages $0.42$ F1 versus $0.35$ for MV-Plurality (a $22\%$ relative gain) and attains the highest F1 on \emph{every} test set once the flip rate exceeds $0.4$
| Subjects: | Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Logic in Computer Science (cs.LO) |
| Cite as: | arXiv:2608.04190 [cs.AI] |
| (or arXiv:2608.04190v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04190 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mario Leiva [view email]
[v1]
Tue, 4 Aug 2026 19:46:57 UTC (6,917 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.