Response Caching

The gateway can answer a request from cache instead of calling your LLM provider. Caching is off unless you ask for it, one request at a time, with the X-PromptGuard-Cache header.

Value	What it does
`exact`	Reuses a response only for an identical request from the same end user.
`semantic`	Also reuses a response for a reworded conversation with every other setting the same.
`off`, or no header	No caching. The request goes to your provider.

Any other value is refused with a 400 and the error code invalid_header.

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: "https://api.promptguard.co/api/v1",
  defaultHeaders: {
    "X-API-Key": process.env.PROMPTGUARD_API_KEY,
    "X-End-User": currentUser.id,
    "X-PromptGuard-Cache": "exact",
  },
});

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="https://api.promptguard.co/api/v1",
    default_headers={
        "X-API-Key": os.environ["PROMPTGUARD_API_KEY"],
        "X-End-User": current_user.id,
        "X-PromptGuard-Cache": "exact",
    },
)

Who can receive a cached response

A cached response goes back only to the end user it was generated for. Requests with the same X-End-User value share a cache, and requests without one share theirs. Send X-End-User when you turn caching on, or every user of your app will share one cache.

What counts as the same request

All of the request is compared, not only the text. A different temperature, max_tokens, response_format, system prompt or tool list never matches. stream is the one exception: it changes how the answer is delivered, not what it is. semantic lets the conversation itself differ in wording. Everything else in the request must still be identical. Use it where a near-paraphrase deserves the same answer, such as a FAQ assistant. Avoid it where one word changes the meaning, such as “delete my account” and “don’t delete my account”. Leave caching off for a request that should give a fresh answer each time, such as a “regenerate” button.

When a request is never cached

These requests always go to your provider, even if they ask for caching:

Streaming requests.
Requests to a project with zero retention turned on. Keeping a response would break the promise that nothing is kept.
Requests whose PII was tokenised. PromptGuard puts the original values back into the response, and those values are not stored.

Seeing what happened

When a request asks for caching, its response carries X-PromptGuard-Cache-Status:

Value	Meaning
`HIT`	Served from cache. Your provider was not called.
`MISS`	Not in the cache. Your provider answered, and the response is now cached.
`BYPASS`	Caching was refused, for one of the reasons above. Your provider answered.

A request that did not ask for caching gets no X-PromptGuard-Cache-Status header.

Limits

Cached responses are kept for one hour for exact and 24 hours for semantic. The cache lives in each gateway instance’s memory, so a request routed to a different instance can miss, and a restart empties it. Responses larger than 1 MB are not cached.

Get Started

Core Concepts

Gateway

Shadow AI

Guides

Cookbooks

Platform

Going to Production

Resources

Roadmap / Preview

Who can receive a cached response

What counts as the same request

When a request is never cached

Seeing what happened

Limits

​Who can receive a cached response

​What counts as the same request

​When a request is never cached

​Seeing what happened

​Limits

Who can receive a cached response

What counts as the same request

When a request is never cached

Seeing what happened

Limits