HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
Quick Answer
This paper shows that HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs.
Quick Take
It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.
Key Points
- HOBA decouples strategic reasoning, model selection, and bid execution across three time scales.
- A large language model infers hyperparameters through a Think-Act-Observe-Reflect loop.
- The SARSA agent incorporates causal adjustment to eliminate selection bias among expert models.
- Dynamic expert pool includes PID, MPC, IQL, and Decision Transformer for bid execution.
- HOBA achieved a +3.6% increase in target cost during large-scale online deployment.
DeepSignal Analysis
What happened
The paper introduces HOBA, a hierarchical reinforcement learning framework designed to enhance online advertising bidding systems. It addresses issues of adaptability in non-stationary auction markets and reduces the need for manual hyperparameter tuning. HOBA employs a large language model for hyperparameter inference and a SARSA agent for expert model selection.
Key evidence
- HOBA utilizes a large language model to infer hyperparameters from contextual signals, improving adaptability in online advertising.
- The framework includes a SARSA agent that dynamically selects among expert models, which helps eliminate selection bias.
- In large-scale deployment, HOBA achieved a +3.6% increase in target cost, demonstrating its effectiveness compared to state-of-the-art baselines.
Why it matters
The introduction of HOBA could significantly impact online advertising by providing a more adaptive and efficient bidding strategy. By reducing the reliance on manual hyperparameter tuning, it lowers operational costs and enhances performance in dynamic auction environments. The reported improvements over existing methods suggest that HOBA may set a new standard in the field.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Online advertising bidding systems typically deploy multiple offline-trained expert models (e.g., PID controllers, model predictive control, offline RL policies) but face two critical limitations: lack of online adaptability to non-stationary auction markets, and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints. We propose HOBA (Hierarchical On-policy Bidding Agents), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales. At the high level, a large language model infers hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop with historical experience retrieval. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias. At the low level, a dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability. Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines. In a large-scale online deployment, HOBA delivered substantial business value, achieving a +3.6\% increase in target cost, proving the effectiveness of our hierarchical multi-agent bidding paradigm.
| Comments: | 10pages,accepted by KDD 2026 ads track |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.24779 [cs.AI] |
| (or arXiv:2607.24779v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24779 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.1145/3770855.3818435
DOI(s) linking to related resources |
Submission history
From: Yunshan Peng [view email]
[v1]
Wed, 17 Jun 2026 15:00:34 UTC (294 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.