AI Gateway adds confidence-based decision fallbacks
Quick Answer
AI Gateway now allows escalation to a fallback model based on confidence conditions, enhancing decision-making reliability.
Quick Take
Users can set conditions for Choice and Score questions, with the ability to combine multiple signals for escalations. This feature is integrated into the existing model framework without disrupting current fallback mechanisms.
Key Points
- Escalation occurs when primary model confidence falls below set thresholds.
- Supports combining multiple conditions for more nuanced decision-making.
- Existing fallback models remain unaffected by new confidence conditions.
- Both stages of decision-making are billed when a fallback is triggered.
- Documentation available for implementing decision fallbacks effectively.
📖 Reader Mode
~2 min readAI Gateway can now escalate a decision request to a fallback model when the primary model's answer trips a confidence condition you set.
Conditions can be combined, so an escalation can depend on more than one signal. The plain model names you already list in models keep catching outright errors, so existing fallbacks are unaffected. Confidence conditions cover Choice and Score questions. Boolean questions escalate on a probability range instead.
Set the fallback by adding one conditional object to providerOptions.gateway.models. Requests without it keep their existing behavior.
import { gateway } from '@ai-sdk/gateway';
import { experimental_decide as decide } from 'ai';
const result = await decide({
model: gateway.decisionModel('typesafe-ai/jev'),
state: 'I was charged twice and now the app will not load.',
questions: {
intent: {
type: 'choice',
instructions: 'Which team should handle this?',
criteria: { billing: 'Charges and refunds', technical: 'Bugs and outages' },
},
},
providerOptions: {
gateway: {
models: [{ model: 'openai/gpt-6-astra', when: { question: 'intent', confidenceBelow: 0.6 } }],
},
},
});
console.log(result.answers.intent.choice);
Escalating an uncertain answer to a fallback model with a confidence condition
Leave out question and the condition checks every question of the matching type, so { confidenceBelow: 0.6 } escalates when any Choice or Score answer falls below 0.6.
Because a triggered fallback runs a second decision, it bills both stages.
Learn more in the decision fallbacks documentation, or see them in practice for mapping CSV columns and flagging UI copy for translation review.
— Originally published at vercel.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from Vercel AI
See more →
The Agent Stack
The Agent Stack by Vercel AI provides essential building blocks for creating production-grade agents, enabling seamless integration across multiple AI models and secure operations. It features components like AI Gateway for model routing, Workflow SDK for durable execution, and Vercel Connect for scoped access, streamlining agent development and deployment across various platforms.
