> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptguard.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Latency Budgets

> Per-detector p99 latency budgets, the three-tier detection path, and how to keep the guard call fast in production.

<Info>
  PromptGuard runs detectors in three tiers: a **fast path** of in-process regex and heuristic checks that covers roughly 95% of traffic in single-digit milliseconds, an **ML path** that calls hosted classifier models, and a **slow path** of LLM-judge detectors that are opt-in and bounded by hard timeouts. This page gives a realistic p99 budget for each detector so you can reason about the tail.
</Info>

## The overall target

The number reported as `processing_time_ms` on each event (see the [Analytics Cookbook](/platform/analytics-cookbook#latency-p50-p95-p99)) is **engine-only** time — it excludes the upstream provider call on the proxy path.

| Path                               | What runs                                       | p99 budget                                                 |
| ---------------------------------- | ----------------------------------------------- | ---------------------------------------------------------- |
| **Fast path (cache hit)**          | Verdict reused from the policy cache            | \< 1 ms                                                    |
| **Fast path (regex + heuristics)** | All deterministic detectors, no network         | a few ms                                                   |
| **ML path**                        | Fast path + one or more hosted classifier calls | tens–low-hundreds of ms (network-bound)                    |
| **Slow path**                      | Fast path + an LLM-judge detector               | bounded by that detector's timeout (6–10 s), **fail-open** |

The design goal is that the **common case stays on the fast path**. ML and LLM detectors only run when a project enables them, when the fast path is ambiguous, or when a plan tier unlocks them — so the tail is opt-in, not paid on every request.

## Fast path — deterministic detectors

These run in-process on every eligible request, in the order below (the engine short-circuits and returns as soon as a blocking detector fires). No network calls, so latency is CPU-bound and scales with input length.

| Detector                  | What it does                                                           | p99 budget |
| ------------------------- | ---------------------------------------------------------------------- | ---------- |
| Policy cache lookup       | Reuse a prior verdict for identical input                              | \< 0.1 ms  |
| Injection (regex layer)   | Pattern rules for instruction-override / role-confusion                | \< 1 ms    |
| Data exfiltration         | System-prompt / training-data extraction patterns                      | \< 1 ms    |
| Fraud / abuse             | Social-engineering and financial-fraud patterns                        | \< 1 ms    |
| Malware                   | Destructive-command and payload patterns                               | \< 1 ms    |
| URL filter                | Allow/block-list, CIDR, scheme, embedded-credential checks             | \< 1 ms    |
| Malicious entity          | Private/obfuscated IPs, homograph domains, shorteners; optional defang | 1–2 ms     |
| PII (regex + checksum)    | 39+ entity types with Luhn/Mod-11/Verhoeff/Mod-97 validation           | 1–3 ms     |
| Encoded-PII               | PII hidden in base64 / hex / URL-encoding                              | 1–2 ms     |
| API-key / secret-key      | Entropy scoring + 40+ known provider prefixes                          | 1–2 ms     |
| Toxicity (regex fallback) | Keyword/category rules when ML is off or down                          | \< 1 ms    |

**Fast-path total p99: a few milliseconds** for typical prompt sizes. Very large inputs (long documents, tool outputs) push the regex and PII stages higher — budget generously if you scan multi-KB payloads.

## ML path — hosted classifier models

When `ML_INFERENCE_MODE=api`, injection, toxicity, and NER-based PII delegate to hosted transformer models. Latency is dominated by the network round-trip and possible model cold-start, **not** local compute.

| Detector       | Default model                                   | p99 budget     | Notes                                                           |
| -------------- | ----------------------------------------------- | -------------- | --------------------------------------------------------------- |
| Injection (ML) | `protectai/deberta-v3-base-prompt-injection-v2` | 50–250 ms warm | 503 on cold-start → auto-retry, then falls back to regex        |
| Toxicity (ML)  | `unitary/toxic-bert`                            | 50–250 ms warm | 0.7 default threshold, 512 max tokens; regex fallback on outage |
| PII NER        | `dslim/bert-base-NER`                           | 50–250 ms warm | Adds `PERSON` / `LOCATION` beyond pattern matching              |

<Note>
  The ML path **fails open to regex**. If the inference API times out, cold-starts, or is unreachable, the request still gets the deterministic verdict — you lose recall, not availability. Flip `ML_INFERENCE_MODE=off` to force regex-only (used for emergency cost/latency cutting). Self-hosted deployments can point `DETECTION_ML_BASE_URL` at a local TGI/vLLM endpoint to keep this path in-network.
</Note>

## Slow path — LLM-judge detectors

These call a generative LLM to reason about context. Each is **opt-in per project or plan tier** and guarded by a hard timeout; on timeout the detector **falls open** and the request proceeds on the fast/ML verdict. Treat the timeout as the p99 ceiling — the budget is "as fast as the model answers, never longer than this."

| Detector                              | Default model                                                            | Timeout (p99 ceiling) | Env override           |
| ------------------------------------- | ------------------------------------------------------------------------ | --------------------- | ---------------------- |
| LLM Guard (custom NL rules)           | `Qwen/Qwen2.5-7B-Instruct`                                               | 8 s                   | `LLM_GUARD_TIMEOUT`    |
| Agentic evaluator                     | `ibm-granite/granite-guardian-3.3-8b` (or `meta-llama/Llama-Guard-3-8B`) | 8 s                   | —                      |
| LLM PII redactor (context-aware pass) | instruct model                                                           | 6 s                   | `LLM_REDACTOR_TIMEOUT` |
| Multi-turn escalation                 | LLM judge                                                                | 8 s                   | —                      |
| Hallucination / groundedness (RAG)    | LLM judge                                                                | 10 s                  | —                      |
| Multimodal OCR extraction             | OCR subprocess                                                           | 8 s                   | —                      |

<Warning>
  A non-instruct **reasoning** model on the LLM-guard slow path can blow the budget: it spends the token allowance "thinking" and gets truncated before emitting the verdict JSON, which then fails open. The default (`Qwen2.5-7B-Instruct`) is a non-thinking instruct model chosen precisely to answer within the 8 s budget. If you override `LLM_GUARD_MODEL`, pick an instruct model.
</Warning>

## Keeping the tail small

* **Lean on the cache.** Identical inputs reuse the prior verdict in \< 0.1 ms. High cache-hit rates are the single biggest lever on p99.
* **Only enable the slow path where it earns its keep.** LLM-judge detectors are for ambiguous, high-stakes surfaces (agentic tool calls, custom NL policies, RAG grounding) — not every endpoint.
* **Right-size `ML_INFERENCE_MODE`.** `api` for recall, `off` for the lowest, most predictable latency, a local `DETECTION_ML_BASE_URL` for in-network ML.
* **Set `fail_mode` deliberately.** `open` favors availability (allow on engine error); `closed` favors safety (block on error). Zero-trust projects should run `closed` and accept that a slow/unreachable detector then blocks rather than falls open.
* **Measure, don't guess.** These are budgets. Read your actual percentiles from the `promptguard.detector.latency` OTEL histogram or the [Analytics Cookbook](/platform/analytics-cookbook#latency-p50-p95-p99). Per-detector timings are also emitted in each event's `event_metadata`.

## Next steps

<CardGroup cols={2}>
  <Card title="Analytics Cookbook" icon="database" href="/platform/analytics-cookbook">
    Measure your real p50/p95/p99 by detector and surface
  </Card>

  <Card title="Threat Detection" icon="shield-halved" href="/security/threat-detection">
    What each detector catches
  </Card>

  <Card title="Reliability" icon="heart-pulse" href="/production/reliability">
    Fail-open vs fail-closed and graceful degradation
  </Card>

  <Card title="Rate Limits" icon="gauge" href="/production/rate-limits">
    Throughput and quota behavior
  </Card>
</CardGroup>