> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptguard.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Fallback Models

> Name the models to try when the one you asked for is rate limited or down, and see which one answered

A request can name up to 3 other models to try, in order, when the model it asks for fails with an error worth retrying. Name them in the `fallback_models` body field, or in the `X-PromptGuard-Fallback-Models` header as a comma-separated list. If you send both, the body field is used. To give every matching request the same chain without changing your code, set it on a [routing rule](#a-chain-on-a-routing-rule) instead.

```javascript theme={"system"}
const completion = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Summarise this ticket." }],
  // @ts-expect-error -- a PromptGuard field, not an OpenAI one
  fallback_models: ["gpt-4o-mini", "claude-3-5-haiku-latest"],
});
```

```python theme={"system"}
completion = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarise this ticket."}],
    extra_body={"fallback_models": ["gpt-4o-mini", "claude-3-5-haiku-latest"]},
)
```

```bash theme={"system"}
curl https://api.promptguard.co/api/v1/chat/completions \
  -H "X-API-Key: $PROMPTGUARD_API_KEY" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "X-PromptGuard-Fallback-Models: gpt-4o-mini, claude-3-5-haiku-latest" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Summarise this ticket."}]}'
```

The field is removed before the request is forwarded, so your provider never receives it.

## When the next model is tried

The next model is tried only after a failure that is worth retrying: a rate limit (`429`), a timeout, a dropped connection or a server error (`5xx`) from the provider, unless the provider says not to retry. Before moving to the next model, PromptGuard tries the same model on the other providers that serve it, and, for a request that is not streamed, on a routing rule's `failover_provider`, as it does for every request.

Any other error, such as a `400` for a malformed request or a `401` for a bad provider key, is returned to you as it is, and no fallback is tried. Sending the same mistake to another model would cost you a second call and give you the same error.

If every model fails, you get the last failure, with that provider's status code. That is true of a streamed request too: you get the error as JSON, not an event stream.

## Streamed requests

A streamed request (`"stream": true`) falls back only **before the first byte has been sent to you**. PromptGuard reads the provider's whole stream, and scans it, before it sends you any of it. So every failure along the way, a rate limit, a server error, a timeout, or a connection that drops before or partway through the provider's answer, happens before your first byte, and the next model answers the stream instead. You never see the failure.

An error the provider reports **inside** a stream it finished, such as an `error` event after some content, is part of that stream: you get it unchanged, and no other model is tried.

When a connection drops partway through the provider's answer and no other provider or fallback model is left to try, you get a `502` with the error code `upstream_interrupted`, and `X-PromptGuard-Model-Used` names the model whose answer broke off. Nothing was sent to you, so retry the request. This applies to requests that are not streamed too. A provider that stops sending partway through, rather than dropping the connection, has timed out, and that is handled like any other timeout.

```json theme={"system"}
{
  "error": {
    "message": "The provider's answer from gpt-4o broke off before it was complete, and no other provider or fallback model was left to try. Nothing was sent to you. Retry the request.",
    "type": "server_error",
    "code": "upstream_interrupted",
    "param": null
  }
}
```

## A chain on a routing rule

A [routing rule](/gateway/routing-rules#fallback-models) can carry the chain in its action, as `"fallback_models"`, so every request it matches falls back with no header and no change to your code:

```json theme={"system"}
{
  "name": "keep GPT-4o answering",
  "condition": { "model": "gpt-4o" },
  "action": { "fallback_models": ["gpt-4o-mini", "claude-3-5-haiku-latest"] }
}
```

A chain in the request replaces the rule's: a request that sends `fallback_models` or `X-PromptGuard-Fallback-Models` uses its own chain, and the rule's models are not tried. Everything else on this page applies to a rule's chain as it does to a request's, with one difference. A request that names a model it may not fall back to is refused, because you sent it and can fix it. A rule's model that a request may not fall back to, because no key is stored for its vendor or because the endpoint's body is another vendor's format, is **skipped** for that request instead: the request goes ahead with its own model and the rule's other fallback models. A rule never stops a request that would otherwise have been answered.

Saving a rule warns you, in the dashboard and in the API's `warnings` field, about each fallback model on a vendor with no stored provider key. The rule is saved anyway, since you may store the key afterwards. Self-hosted deployments count skipped fallback models in `promptguard_fallback_models_skipped_total` on `/metrics`, by `reason`.

## One time limit for the whole request

Every call PromptGuard makes to a provider for one request shares one time limit: the same limit as a single call to your provider, counted from when the first call is made. That covers the other providers tried for the same model, a routing rule's `failover_provider`, and every model in the chain. Each call gets only the time left, so a request never takes longer than one call could. When too little time is left for another call to answer, it is not made, and you get the last answer a provider gave, with `X-PromptGuard-Model-Used` naming the model that gave it.

If no provider answered at all in that time, you get a `504` with the error code `upstream_timeout`:

```json theme={"system"}
{
  "error": {
    "message": "No provider answered within 570 seconds, the time limit this request's upstream attempts share (provider failover and fallback models included). Retry with backoff; with a fallback chain, put the faster models first.",
    "type": "server_error",
    "code": "upstream_timeout",
    "param": null
  }
}
```

So a model that is slow to fail leaves less time for the next. If you expect long completions, put the fastest models first.

## Which vendors a fallback can use

A fallback model can be on the **same vendor** as the model you asked for, or on a vendor you hold a **[stored provider key](/gateway/provider-keys)** for: a key stored for this project or for your account. These are the same keys PromptGuard uses for the model you asked for. Storing a key is how you agree to send that vendor your prompts. Each fallback is sent with its own vendor's stored key, never with the key meant for another vendor.

If a fallback is on any other vendor, the whole request is refused with a `400` before anything is sent. The error code is `fallback_model_not_allowed`, and the message names the model, its vendor and the reason:

```json theme={"system"}
{
  "error": {
    "message": "Fallback model 'claude-3-5-haiku-latest' is served by anthropic, and no anthropic provider key is stored for this project or your account. Store one to allow falling back to anthropic.",
    "type": "invalid_request_error",
    "code": "fallback_model_not_allowed",
    "param": "fallback_models"
  }
}
```

A fallback on another vendor works on `/api/v1/chat/completions` only. On `/api/v1/messages` and `/api/v1/responses` the request body is in one vendor's own format, so a fallback there must be on that vendor too. A fallback on another vendor there is refused with the same `fallback_model_not_allowed`, and the message says why.

A fallback model that no provider serves is refused with a `404` and the error code `model_not_found`.

## Which model answered

Every response from your provider, streamed or not, success or error, carries an `X-PromptGuard-Model-Used` header naming the model that produced it. When a fallback answered, the header names the fallback. When no fallback was needed, it names the model you asked for, or the model a [routing rule](/gateway/routing-rules) replaced it with.

For a response that is not streamed, the security event for the response records that model and the token usage it reported, so your dashboard shows the model you were actually billed for.

A response that was served from the [cache](/gateway/caching), or that PromptGuard blocked or refused before calling your provider, has no `X-PromptGuard-Model-Used` header: no provider was called for it. A fallback's answer is never cached.

Browser code on an allowed origin can send `X-PromptGuard-Fallback-Models` and read `X-PromptGuard-Model-Used`.

## Limits

* At most 3 fallback models per request. More is refused with a `400` and the error code `invalid_fallback_models`. So is a `fallback_models` value that is not a list of model names. A routing rule takes at most 3 as well; saving one with more is refused.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.