What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation
Quick Answer
This study introduces a three-class detection framework for distinguishing humans, bots, and AI agents, revealing that binary classifiers misclassify 39.1% of AI agents as humans.
Quick Take
By incorporating an explicit agent class, the framework achieves a perfect F1 score for agent detection across multiple runs. Key behavioral features like mouse_event_rate and teleport_click_ratio enable robust detection, achieving 100% recall at every evasion level.
Key Points
- Binary classifiers misclassify 39.1% of AI agents as humans, highlighting architectural limitations.
- Incorporating an agent class yields perfect F1 scores across 30 runs with various models.
- Two features, mouse_event_rate and teleport_click_ratio, achieve 100% agent recall at all evasion levels.
- Five features improve macro-F1 to 0.991, effectively distinguishing all traffic classes.
- The study constructs a five-level evasion ladder to measure detection robustness.
DeepSignal Analysis
What happened
A new detection framework has been developed to differentiate between humans, bots, and AI agents, addressing limitations in existing binary classifiers. The framework achieved perfect F1 scores for AI agent detection, highlighting the inadequacy of traditional methods in recognizing AI traffic. Key behavioral features were identified that enable robust detection across various evasion techniques.
Key evidence
- The study found that a binary classifier misclassifies 39.1% of AI agents as humans, indicating a significant flaw in current detection methods.
- By introducing an explicit agent class, the new framework achieved an F1 score of 1.000 for agent detection across 30 runs, demonstrating its effectiveness.
- Two behavioral features, mouse_event_rate and teleport_click_ratio, provided 100% recall for agent detection at all evasion levels, showcasing the robustness of the framework.
Why it matters
This research is crucial as it addresses a growing challenge in cybersecurity and AI ethics, where traditional detection systems fail to accurately identify AI agents. The ability to distinguish between human and AI traffic is essential for maintaining the integrity of online interactions and preventing misuse of automated systems. The findings could influence future developments in bot detection technologies.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Bot detectors deployed at scale treat traffic as binary: human or bot. This assumption breaks when AI agents browse the web through browser automation, a traffic class that is neither and that binary classifiers structurally cannot represent. We present a three-class detection framework distinguishing humans, bots, and AI agents, and show that the binary-vs-agent confusion is architectural: a binary human-vs-bot detector misroutes agent sessions because its label space lacks an agent class. On our controlled benchmark, an MLP binary classifier misclassifies 39.1% of real AI agents as human and a SAINT binary transformer misclassifies 34.5%; adding an explicit agent class yields per-class agent F1 = 1.000 in all 30 runs (3 model families $\times$ 10 seeds). To measure evasion resistance, we construct a five-level evasion ladder spanning passive observation, GAN-generated trajectories, and replay of real human cursor data ($n = 2299$ evasion sessions). Across 10 seeds and 3 model families we observe zero agent misses in 22990 per-seed predictions. The discriminative signal is a browser-automation artifact, not evidence of agent reasoning: Playwright does not emit the raw pointer-move and wheel-delta streams a physical input device produces, and this absence signature survives trajectory manipulation. Exhaustive search over all feature subsets of size 1-5 (9401 GBMs) shows that two behavioral features (mouse_event_rate, teleport_click_ratio) give 100% observed agent recall at every evasion level with agent precision 0.994; five features lift macro-F1 to 0.991. The signal is redundantly encoded: removing teleport_click_ratio leaves agent detection at 100%. The single-feature regime is degenerate, flagging every agent only by collapsing the classifier to always predict "agent". Two features robustly isolate agents; five separate all three traffic classes at macro-F1 $\geq 0.99$.
| Comments: | 17 pages (11 main + appendices), 7 figures. Accepted at the North East AI Agents Day 2026 workshop, Jane Street, New York City. Workshop: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| ACM classes: | I.2.6; K.6.5 |
| Cite as: | arXiv:2607.26935 [cs.AI] |
| (or arXiv:2607.26935v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.26935 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Vishisht Choudhary [view email]
[v1]
Wed, 29 Jul 2026 14:05:26 UTC (481 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.