Catching rogue AI behavior with identity-aware analytics

When you look at your AI bill, it can be hard to tell if anything is amiss. You first need a baseline so you can see what has changed, whether it’s an agent that’s gone wild or an employee whose usage has spiked 10x. Being able to spot those shifts lets you start investigating, and so far, it’s been hard to see them.

Knowing who is doing what with AI is one of the key challenges organizations are confronting right now. One report from Stanford University found that 59% of organizations said knowledge gaps were their biggest obstacle to responsible AI governance. 

This is a security problem as much as a financial one. Solving these issues takes two things: a verified identity on every request (so a spike has a name behind it), and a picture of what normal looks like for that identity. Today we're announcing both.

Identity-aware AI Gateway with Cloudflare Access is now in open beta, and User Insights is generally available to every AI Gateway customer at no additional cost. Together they turn the traffic already flowing through AI Gateway into a behavioral baseline for every person and agent using it, and identify the ones that break from it.

What is AI Gateway?

AI Gateway is the central control plane for all of your AI usage. Instead of every app and team calling models on OpenAI, Anthropic, Google, or Workers AI directly, requests route through AI Gateway first, giving you one place to observe, secure, and govern all your AI usage.

It works with the applications you build, and with the coding tools your developers already live in. Route agent harnesses like Claude Code, Codex, and GitHub Copilot through AI Gateway, and they fall under the same visibility and controls as everything else.

Identity-aware AI Gateway

With the AI Gateway and Cloudflare Access integration, you can put a custom domain in front of your gateway and protect it with Access, just like any other application. That means you can:

• Authenticate with any SAML-supported identity provider, like Okta or Entra, removing the need to generate and pass around Cloudflare API keys.
• Set policies on exactly who can access your gateway.
• Send requests to a clean hostname like ai.example.com, with no account ID or gateway ID in the URL.

Every authenticated request now carries the user's identity from Access. AI Gateway adds the verified Access user ID to request metadata as cf.user_id, so you can filter logs, analytics, and spend by the person who actually made the request.

Coupled with spend limits, that identity becomes a budgeting tool. Because each request now carries a real user, you can set per-user spend limits: give every user their own budget bucket, then block further requests or fall back to a cheaper model when they hit it. No more surprise invoices, and no shared API key hiding who spent what.

One of our early adopters, Flexport, ran into exactly this problem.

"Shared API keys make it almost impossible to tell who is using an AI service or apply the access rules we already have for employees,” says Max Baumgarten, Staff Security Engineer at Flexport. “Putting Cloudflare Access in front of AI Gateway gives each request an authenticated identity and lets us use our existing identity policies at the gateway. Our teams can adopt AI tools without creating a separate authentication system for every client."

In the near future, you'll be able to use your users' identity provider groups to set spend limits or control which models a group can access. For example, give your machine learning team access to frontier models, cap the spend of your support team, or scope a budget to everyone working on a specific project, all mapped to the groups you already manage in your identity provider.

The new User Insights tab

Within AI Gateway, you will now see a tab called User Insights. User Insights reads the traffic passing through your gateway and turns it into a behavioral picture of every account. It learns how each account normally acts, identifies the ones that break from that pattern, and gives you the context to tell a rogue agent from a busy engineer. It works on the traffic already going through your gateway, so there's nothing to set up.

User Insights tracks cost, including where it's being wasted, such as low cache-hit rates and oversized context windows. Plenty of tools already do that. What they don't do is tell you whether an account is behaving normally. That's what we chose to focus on, alongside cost controls. 

Baselining every account: people and agents

Every account leaves a behavioral fingerprint over time, whether it's a person or agent. An agent summarizing tickets every three hours is tight and consistent. A person is messier, with varied prompts, irregular timing, and long sessions on hard problems. Both are legitimate, so the same deviation can be noise for one and a real signal for the other.

In User Insights, we start by scoring sessions, not single requests. Absolute thresholds fail here: a $500 jump from a heavy user might be normal, while a $50 session from an agent that always spends $5 is a 10x change that could otherwise slip by. So we compare each session against the account's own history, using its 95th percentile (p95) session cost over the last 30 days. That gives us a read on how the account normally operates, and anything above 2x of its p95 is a strong candidate for anomalous behavior.

The following analysis outlines how we arrived at these numbers.

Figure 1: Session Cost Anomaly Detection

How to read the chart above 

The chart plots real sessions from our own internal traffic. Each point represents an individual session (plotted on log scales):

• X-axis (Session Cost): Total cost in dollars.
• Y-axis (x User p95): How many times the session exceeded the user's personal baseline.

The two dashed threshold lines divide the sessions into four categories:

• Top-Right (★ Stars): Exceeds both the 2x user p95 baseline and the account-level p99 ceiling. These are high relative spikes that represent meaningful abnormal spend and will trigger an alert. 
• Top-Left: High relative spike (2x user p95), but below the account p99 floor. We ignore this to avoid alerting on small-dollar shifts.
• Bottom-Right: High absolute spend, but consistent with this user's typical high usage. This is also ignored as routine behavior.
• Bottom-Left: Normal activity well within both baselines.

Figure 2: Account-level Session Cost Distribution

This histogram (Figure 2) maps every session cost across the organization to establish an account-wide ceiling:

• Typical Usage: The vast majority of sessions cost well under $10, with the 95th percentile sitting at $20.
• Account p99 ($200): Only 1% of all sessions across the entire company reach or exceed $200.

So why did we pick p99? Setting our absolute dollar ceiling at the account p99 creates a meaningful bar. It guarantees that an anomaly isn't just a sudden shift for one specific user, but also ranks among the most expensive 1% of sessions across the entire organization.

Figure 3: Single User Session History

Baselines aren't static. As an account's habits change, its rolling p95 (green line) and 2x threshold (orange line) move with it, so an alert always reflects recent behavior rather than a number set once. We also apply a dollar floor so that a spike has to be both statistically unusual and worth an admin’s time to investigate. That dollar floor is what keeps a micro-user's 500x blip over a few cents from ever firing an alert.

The right lens for detecting rogue behavior 

After all the analysis above, what admins see is a view of the accounts that broke their own pattern with everything normal filtered out. That filtered view is a rogue behavior feed.

This behavior is hard to catch because the signal is never a new tool or a blocked action. It's a trusted account doing more of what it's already allowed to do. It might be a service account that suddenly starts running more expensive sessions, or a person whose usage jumps well past their own norm and stays there for days.

None of these trip a policy, but all of them break a behavioral baseline. A sudden departure from an account's own usage is often the first observable sign of a compromised credential or an agent going off the rails.

User Insights does not decide intent and it does not block anyone; instead, it puts the handful of accounts that started behaving strangely in front of an admin so someone can ask the next question. Sometimes that leads to a real investigation. Sometimes it just means that someone needs coaching (like the developer who dumps a whole codebase into every prompt when a snippet would do). 

What's next 

We’ll help you move from cost control to cost optimization

Once you’ve set a budget, the natural next question is: how can you get the equivalent output quality at lower cost? Not every request needs a frontier model. A summarization task or a simple code completion can run on a cheaper model without meaningful quality loss.

We're building task-based smart routing, where AI Gateway analyzes the incoming request and routes it to the model that gives you the best result at the lowest cost. At the organizational level, you’ll be able to see where you can capture the most savings by routing to more efficient models.Task-based smart routing is in active development. We'll share more as it matures.

We’ll help you understand how AI is being used

Anomaly detection tells you an account broke its pattern, but not why. An admin still has to dig into the logs and piece together what happened. Closing that gap is what we're focused on next, and it starts with classifying what the traffic actually is.

We're building prompt classification that sorts requests into categories like coding, writing, and others. These categories are  the context missing from almost every other signal. A spend spike in “coding” from an engineer might be acceptable, but the same spike in a category that account has never touched is not. Classification can show an organization not just how much AI it uses, but what it uses AI for. 

It also answers the question underneath most of these conversations: is AI being used for the work it was intended? Once business traffic is separated from everything else, personal use becomes visible. From the outside, someone running a side hustle on company time and someone quietly moving data out through a model look the same. Telling them apart is central to catching insider risk. 

Once your AI traffic is running through AI Gateway, each new category of risk or efficiency signal is one more thing an admin gets with no extra setup.

Get started

User Insights is generally available today to every AI Gateway customer at no additional cost. It's already in the dashboard for anyone sending traffic through the gateway, so if you're already routing through AI Gateway, this view is available to you. 

If you haven't already, create a gateway and start making requests to any model in our catalog. 

We recommend that you put AI Gateway behind Cloudflare Access which is now in open beta. The spend and anomaly views work without it, but attaching an identity is what turns an anonymous account ID into a name you can actually act on. Start in monitoring mode to learn your baselines before you enforce anything.

We want to hear how you're managing AI today. Join the conversation on Discord or reach out to your account team.