fallback_models body field, or in the X-PromptGuard-Fallback-Models header as a comma-separated list. If you send both, the body field is used. To give every matching request the same chain without changing your code, set it on a routing rule instead.
When the next model is tried
The next model is tried only after a failure that is worth retrying: a rate limit (429), a timeout, a dropped connection or a server error (5xx) from the provider, unless the provider says not to retry. Before moving to the next model, PromptGuard tries the same model on the other providers that serve it, and, for a request that is not streamed, on a routing rule’s failover_provider, as it does for every request.
Any other error, such as a 400 for a malformed request or a 401 for a bad provider key, is returned to you as it is, and no fallback is tried. Sending the same mistake to another model would cost you a second call and give you the same error.
If every model fails, you get the last failure, with that provider’s status code. That is true of a streamed request too: you get the error as JSON, not an event stream.
Streamed requests
A streamed request ("stream": true) falls back only before the first byte has been sent to you. PromptGuard reads the provider’s whole stream, and scans it, before it sends you any of it. So every failure along the way, a rate limit, a server error, a timeout, or a connection that drops before or partway through the provider’s answer, happens before your first byte, and the next model answers the stream instead. You never see the failure.
An error the provider reports inside a stream it finished, such as an error event after some content, is part of that stream: you get it unchanged, and no other model is tried.
When a connection drops partway through the provider’s answer and no other provider or fallback model is left to try, you get a 502 with the error code upstream_interrupted, and X-PromptGuard-Model-Used names the model whose answer broke off. Nothing was sent to you, so retry the request. This applies to requests that are not streamed too. A provider that stops sending partway through, rather than dropping the connection, has timed out, and that is handled like any other timeout.
A chain on a routing rule
A routing rule can carry the chain in its action, as"fallback_models", so every request it matches falls back with no header and no change to your code:
fallback_models or X-PromptGuard-Fallback-Models uses its own chain, and the rule’s models are not tried. Everything else on this page applies to a rule’s chain as it does to a request’s, with one difference. A request that names a model it may not fall back to is refused, because you sent it and can fix it. A rule’s model that a request may not fall back to, because no key is stored for its vendor or because the endpoint’s body is another vendor’s format, is skipped for that request instead: the request goes ahead with its own model and the rule’s other fallback models. A rule never stops a request that would otherwise have been answered.
Saving a rule warns you, in the dashboard and in the API’s warnings field, about each fallback model on a vendor with no stored provider key. The rule is saved anyway, since you may store the key afterwards. Self-hosted deployments count skipped fallback models in promptguard_fallback_models_skipped_total on /metrics, by reason.
One time limit for the whole request
Every call PromptGuard makes to a provider for one request shares one time limit: the same limit as a single call to your provider, counted from when the first call is made. That covers the other providers tried for the same model, a routing rule’sfailover_provider, and every model in the chain. Each call gets only the time left, so a request never takes longer than one call could. When too little time is left for another call to answer, it is not made, and you get the last answer a provider gave, with X-PromptGuard-Model-Used naming the model that gave it.
If no provider answered at all in that time, you get a 504 with the error code upstream_timeout:
Which vendors a fallback can use
A fallback model can be on the same vendor as the model you asked for, or on a vendor you hold a stored provider key for: a key stored for this project or for your account. These are the same keys PromptGuard uses for the model you asked for. Storing a key is how you agree to send that vendor your prompts. Each fallback is sent with its own vendor’s stored key, never with the key meant for another vendor. If a fallback is on any other vendor, the whole request is refused with a400 before anything is sent. The error code is fallback_model_not_allowed, and the message names the model, its vendor and the reason:
/api/v1/chat/completions only. On /api/v1/messages and /api/v1/responses the request body is in one vendor’s own format, so a fallback there must be on that vendor too. A fallback on another vendor there is refused with the same fallback_model_not_allowed, and the message says why.
A fallback model that no provider serves is refused with a 404 and the error code model_not_found.
Which model answered
Every response from your provider, streamed or not, success or error, carries anX-PromptGuard-Model-Used header naming the model that produced it. When a fallback answered, the header names the fallback. When no fallback was needed, it names the model you asked for, or the model a routing rule replaced it with.
For a response that is not streamed, the security event for the response records that model and the token usage it reported, so your dashboard shows the model you were actually billed for.
A response that was served from the cache, or that PromptGuard blocked or refused before calling your provider, has no X-PromptGuard-Model-Used header: no provider was called for it. A fallback’s answer is never cached.
Browser code on an allowed origin can send X-PromptGuard-Fallback-Models and read X-PromptGuard-Model-Used.
Limits
- At most 3 fallback models per request. More is refused with a
400and the error codeinvalid_fallback_models. So is afallback_modelsvalue that is not a list of model names. A routing rule takes at most 3 as well; saving one with more is refused.