Failover
Use provider failover to keep LLM requests running when a provider is unavailable.
When a provider cannot serve a request, AllModels can try another eligible provider for the same model before output begins.
When failover happens
With fallbacks enabled, AllModels can try another provider when the selected provider:
- Cannot be reached or times out
- Rate-limits the request
- Returns a server error
- Does not support an input modality used by the request, such as images
Invalid requests and other client errors are returned immediately. Once a streaming response has begun, it is not replayed through another provider.
Control failover
Fallbacks are enabled by default. Set allow_fallbacks to false when the request must not move beyond the providers you selected:
{
"provider": {
"order": ["YOUR_PROVIDER_ID"],
"allow_fallbacks": false
}
}Use order to express a preference, only to restrict the eligible set, and ignore to exclude a provider. See Providers for all routing options and the LLM model catalog for the providers available for each model.
Image requests
If an image request selects a provider that does not support images, enabled fallbacks allow AllModels to use another provider that does. If none of the permitted providers support the input, the request returns 400 unsupported_input_modality.
When all routes fail
| Status | Code | Meaning |
|---|---|---|
404 | no_matching_provider | Provider preferences excluded every route. |
429 | rate_limit_exceeded | The last eligible provider was rate-limited. |
502 | upstream_error or upstream_unavailable | Every eligible provider failed or was unreachable. |
503 | model_unavailable | No eligible provider is currently available to serve the model. |
Every completion response includes x-request-id, which can be used to trace the request.