BOHM: Zero-Cost Hierarchical Attribution for Compound AI Systems
Quick Answer
BOHM introduces a zero-cost hierarchical attribution method for compound AI systems, outperforming SHAP with a Kendall tau of 0.928 on 18 LLMs while requiring significantly fewer evaluations.
Quick Take
BOHM introduces a zero-cost hierarchical attribution method for compound AI systems, outperforming SHAP with a Kendall tau of 0.928 on 18 LLMs while requiring significantly fewer evaluations. This approach utilizes existing routing weights, enabling multi-resolution attribution without needing access to component internals.
Key Points
- BOHM achieves Kendall tau=0.928 with 18 LLMs, outperforming SHAP's tau=0.980 at 9,000x more evaluations.
- The method requires no access to component internals, leveraging existing routing weights for attribution.
- BOHM provides multi-resolution attribution, unlike flat methods that cannot offer simultaneous evaluations.
- In a US Census hierarchy, BOHM recovers ground-truth rankings with tau up to 0.722.
- The approach satisfies efficiency, monotonicity, symmetry, and weak suppression, but not Shapley's additivity.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 22866v1 Announce Type: new Abstract: Compound AI systems route tasks through hierarchies of specialised components. Attribution is dominated by Shapley-based methods (SHAP), which decompose a coalition value function into per-component marginal contributions and require evaluation of the system on arbitrary component subsets.
That requirement fails for third-party APIs, opaque endpoints, and agentic orchestrators that concentrate routing on a few tools, leaving most coalitions un-evaluable from the deployed orchestrator. We introduce BOHM, which extracts a hierarchical attribution tree directly from the routing weights such systems already maintain: leaf attribution is the path product of root-to-leaf routing weights; level-k attribution is the induced distribution over depth-k nodes.
The method has zero marginal cost, requires no access to component internals, and provides multi-resolution attribution at every level simultaneously, which flat methods cannot offer at any evaluation budget. BOHM and SHAP answer different questions and converge when the deployed router routes near-optimally. On 18 LLMs in a 3-level hierarchy over 880 problems, BOHM yields Kendall tau=0. 928; SHAP reaches tau=0. 980 at 9,000x more coalition evaluations per seed.
On a 5-driver, 7-benchmark agentic study (35 cells, complete coverage), drivers concentrate routing on a single tool (top-share median 0. 65), and cell-level tau(BOHM,SHAP) is predicted by whether the driver's top pick is the empirically best tool (mean +0. 22 vs ~+0. 01). On a US Census hierarchy (475 leaves, 4 levels), BOHM recovers ground-truth rankings at every level (tau up to 0. 722). BOHM satisfies efficiency, monotonicity, symmetry, and weak suppression but not Shapley's additivity.
It is best understood as a complementary primitive: a multi-resolution decomposition computable wherever routing state exists, whose disagreement with Shapley is itself diagnostic.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Information Limits and Attractor Dynamics in Economies of Frontier LLM Agents: A Pre-Registered Test
A pre-registered experiment on Claude Opus 4.8 investigates wealth growth and population misalignment in economies, revealing that relative growth aligns with claimed information but fails to demonstrate expected noise-maintained dispersion. The experiment cost $138.76 and is fully reproducible from cached outputs.