Model routing and failover

Distinguish model IDs, provider selection, channel retries, and application fallback.

How a request is routed

The client sends one deployment-published model ID. The gateway filters candidate channels using the token’s permitted provider category, automatic or manual routing settings, current grants, model, request path, and channel availability. Manual bindings and automatic routing follow their respective configuration; automatic routing does not grant access to every provider.

Concept	What an integrator needs to know
Model ID	Select from the target deployment’s model list; preserve prefixes and case
Model mapping	Operators can map a public ID to an upstream ID; the request ID need not equal the upstream name
Providers and channels	A model can have multiple candidate channels; deployment policies control ordering, weights, health, and capacity
Channel retry	The gateway retries within one request according to policy; budgets, status codes, and channel overrides affect the result
Application model fallback	The application sends a new request using another model; revalidate capabilities, context, and output structure

Visibility in the model list does not prove every channel supports tools, JSON Schema, media, or state recovery. Start with model capabilities. The request’s group is not a general provider-selection field; current code handles it only on the dedicated Playground route.

Retry boundaries

The gateway considers cancellation, retry budgets, error classification, channel constraints, and session policy. Once an ordinary HTTP response has started writing, the corresponding retry path stops; strict sessions or pinned channels can also prevent switching. Do not promise that every failure selects another channel.

Client SDK retries and internal gateway attempts are separate counters. Local examples disable automatic SDK retries for generation. Before production, confirm the internal budget with your operator, then configure application deadlines, queue limits, and recovery policy.

When a backup model is required

1. Validate both models’ protocols, tool schemas, context capacity, and output constraints in advance.
2. Specify eligible failures and a total attempt limit; fix authentication, permission, and parameter errors first.
3. Record the model and gateway request ID for each new request. If output has already been shown, clearly begin a new attempt instead of concatenating outputs into one answer.
4. Check the state of executed tools, submitted media tasks, and external writes first. A network timeout does not prove the upstream did no work.
5. Do not assume Responses IDs, compacted state, native signatures, or WebSocket sessions transfer between channels or models. Use reviewed portable conversation history where appropriate.

This site does not provide a generic automatic generation-resubmission script: the application must handle duplicate work and state compatibility. A read-only retry example is available in rate limits and retries.

Migrating from OpenRouter

OpenRouter’s models fallback array, provider.order, routing suffixes, and dedicated limit-inspection API are outside the general integration contract verified here. Interface compatibility alone does not make those settings portable. These examples use a single model; confirm any deployment-specific extension separately.

Verified against project revision 30a6d88a4: middleware/distributor.go, service/channel_select.go, service/group.go, service/relay_error.go, and relay/helper/model_mapped.go. Operators manage routing settings; this guide does not change administration configuration.

SDK migration · Production integration

阅读中文