A Cost-Aware, Paired Protocol for Auditing Dynamic Tool Synthesis in Agentic Video Question Answering
Quick Answer
The proposed cost-aware, paired protocol audits tool-augmented VideoQA systems, revealing that Dynamic-SAGE improves accuracy by 7.5 points while increasing inference costs by 26%.
Quick Take
The proposed cost-aware, paired protocol audits tool-augmented VideoQA systems, revealing that Dynamic-SAGE improves accuracy by 7.5 points while increasing inference costs by 26%. This method evaluates both accuracy and cost jointly, providing insights into efficiency gains and regressions across different question types.
Key Points
- Dynamic-SAGE improves accuracy by 7.5 points (p < 0.001) over the SAGE baseline.
- Inference costs rise by 26% with a 34% increase in token usage.
- Visible tool calls and reasoning turns are reduced by approximately 28%.
- Performance gains are largest on visual and open-ended questions.
- The protocol uses McNemar's test for significance and paired bootstrap confidence intervals.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2607. 01469v1 Announce Type: new Abstract: Agentic Video Question Answering (VideoQA) systems invoke tools during inference, but their tool libraries are fixed, so recurring procedures are rebuilt from primitives on every question. Synthesizing composite tools could remove this overhead, but whether such expansion helps is hard to assess: final-answer accuracy, the standard metric, ignores inference effort, so it cannot reveal how a system shifts cost.
We propose a cost-aware, paired protocol for auditing tool-augmented video agents. The protocol pairs two complete systems on the same input for each question and reports their net difference across accuracy and cost jointly. For each question, it sorts the paired outcome into one of six groups defined by joint correctness and by the change in visible tool calls, separating accuracy-preserving efficiency gains from harmful regressions.
Significance is reported with McNemar's test and paired bootstrap confidence intervals. We instantiate the protocol on Dynamic-SAGE, an agentic VideoQA framework that synthesizes, validates, and persistently registers executable composite tools for reuse on unseen questions, and evaluate it against the SAGE baseline on SAGE-Bench. The audit reveals a multi-axis profile that a scalar accuracy comparison would miss: Dynamic-SAGE improves accuracy by 7. 5 points (p < 0.
001) and reduces reasoning turns and visible tool calls by roughly 28%, while shifting rather than reducing inference cost, as token usage rises 34% and cost 26%. Gains are largest on visual and open-ended questions and neutral on verbal and multimodal ones, and residual failures concentrate on hard, open-ended questions where the pipeline does the most work. By measuring accuracy and cost jointly, the protocol shows where the pipeline-level difference is reliable and where it is not.
The code is available at https://github. com/KurbanIntelligenceLab/Dynamic-SAGE.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.