AlloyDB Ships Proxy Models That Replace LLM Calls with Local Inference Inside the Database

Google recently announced the general availability (GA) of AlloyDB AI functions, along with two acceleration techniques that fundamentally change how databases interact with large language models (LLMs).

According to Google's internal testing, smart batching delivers a 2,400x improvement in throughput over row-at-a-time processing, while optimized proxy models push that to 23,000x with a 6,000x reduction in cost. While these headline numbers warrant engineering scrutiny, the underlying proxy model architecture is what practitioners should focus on.

With AlloyDB AI functions, developers can call LLMs directly within standard SQL queries. The GA release includes ai.generate for text generation, ai.if for semantic filtering, ai.rank for semantic reranking, ai.forecast for time-series prediction, and three new additions:

• ai.summarize,
• ai.agg_summarize for group-level summaries,
• and ai.analyze_sentiment.

Because these execute as standard SQL operators, a query can filter rows by meaning rather than rigid keyword matches.

The problem with calling an LLM per row is obvious at scale: a table with 100,000 products means 100,000 round trips to Vertex AI, each carrying the same system prompt, each waiting for model inference, and each incurring per-token costs. Two acceleration layers address this.

Smart batching, now available for ai.if and ai.rank, groups multiple rows into a single model call. Instead of sending the system prompt with every row, AlloyDB sends it once and batches the data. Google reports throughput of up to 10,000 rows per second in internal testing, a 2,400x improvement over the row-at-a-time baseline. The technique is straightforward, and the gains are credible for workloads where the per-row payload is small relative to the prompt.

The proxy model is the more architecturally significant move. For ai.if queries (currently in preview), AlloyDB introduces a two-phase workflow:

-- Phase 1: Train the local proxy model using a sample of data and the frontier LLM

PREPARE underwater_suitability_proxy FROM
SELECT description FROM products;

-- Phase 2: Execute the query at database speed using the local proxy model

SELECT * FROM products
WHERE ai.if(description, 'suitable for underwater use deeper than 60 meters')
USING proxy(underwater_suitability_proxy);

First, a PREPARE statement sends a sample of your data to a frontier model and uses the results to train a lightweight local model inside the database. Then, EXECUTE runs the query using the local proxy instead of calling the external LLM. AlloyDB falls back to the frontier model if the proxy's confidence is too low or if no trained model is available. Google reports throughput of 100,000 rows per second with this approach.

The proxy model pattern inverts the usual database-to-LLM relationship. Instead of the database being a client that calls an external model for every decision, the database becomes a student that learns the model's judgment on a sample, then applies that judgment locally at database speed. The LLM becomes a teacher rather than a runtime dependency.

This pattern has implications beyond AlloyDB. Any database that calls external models for per-row decisions faces the same cost and latency wall. The question is whether competitors (Aurora, Azure SQL, CockroachDB, PlanetScale) will adopt similar distillation-at-query-time approaches, or whether they will rely on users building this logic in application code.

Practitioners should note the caveats in Google's own announcement: the 23,000x and 6,000x numbers come from internal testing, apply specifically to ai.if in preview, and do not represent the performance of all AI functions generally. The optimized proxy model is not yet GA. Teams evaluating AlloyDB for production AI workloads should benchmark against their own data distributions and query patterns before committing.

Raimundas Juodvalkis, a Starburst architect, offered practical framing on LinkedIn:

Treat them as governed database extensions, not magic WHERE clauses.

He recommended starting with read-heavy review workflows before writing model-derived fields back into core systems, and tracking model cost separately from query cost.

The release also includes a managed MCP server for AlloyDB, letting AI agents query database content through the Model Context Protocol without teams running and scaling their own MCP infrastructure. Combined with existing vector search using Google's ScaNN index (up to 10 billion vectors), AlloyDB is positioning itself as a PostgreSQL-compatible database where structured queries, semantic search, and LLM-powered analysis coexist in the same SQL layer.

AlloyDB AI functions are generally available on PostgreSQL 17 instances. Smart batching is GA for ai.if and ai.rank. The optimized proxy model for ai.if is in preview. AI function acceleration requires setting a database flag (google_ml_integration.enable_ai_function_acceleration) and is not enabled by default.