Skip to main content
PromptGuard runs detectors in three tiers: a fast path of in-process regex and heuristic checks that covers roughly 95% of traffic in single-digit milliseconds, an ML path that calls hosted classifier models, and a slow path of LLM-judge detectors that are opt-in and bounded by hard timeouts. This page gives a realistic p99 budget for each detector so you can reason about the tail.

​
The overall target

The number reported as processing_time_ms on each event (see the Analytics Cookbook) is engine-only time — it excludes the upstream provider call on the proxy path.
PathWhat runsp99 budget
Fast path (cache hit)Verdict reused from the policy cache< 1 ms
Fast path (regex + heuristics)All deterministic detectors, no networka few ms
ML pathFast path + one or more hosted classifier callstens–low-hundreds of ms (network-bound)
Slow pathFast path + an LLM-judge detectorbounded by that detector’s timeout (6–10 s), fail-open
The design goal is that the common case stays on the fast path. ML and LLM detectors only run when a project enables them, when the fast path is ambiguous, or when a plan tier unlocks them — so the tail is opt-in, not paid on every request.

​
Fast path — deterministic detectors

These run in-process on every eligible request, in the order below (the engine short-circuits and returns as soon as a blocking detector fires). No network calls, so latency is CPU-bound and scales with input length.
DetectorWhat it doesp99 budget
Policy cache lookupReuse a prior verdict for identical input< 0.1 ms
Injection (regex layer)Pattern rules for instruction-override / role-confusion< 1 ms
Data exfiltrationSystem-prompt / training-data extraction patterns< 1 ms
Fraud / abuseSocial-engineering and financial-fraud patterns< 1 ms
MalwareDestructive-command and payload patterns< 1 ms
URL filterAllow/block-list, CIDR, scheme, embedded-credential checks< 1 ms
Malicious entityPrivate/obfuscated IPs, homograph domains, shorteners; optional defang1–2 ms
PII (regex + checksum)39+ entity types with Luhn/Mod-11/Verhoeff/Mod-97 validation1–3 ms
Encoded-PIIPII hidden in base64 / hex / URL-encoding1–2 ms
API-key / secret-keyEntropy scoring + 40+ known provider prefixes1–2 ms
Toxicity (regex fallback)Keyword/category rules when ML is off or down< 1 ms
Fast-path total p99: a few milliseconds for typical prompt sizes. Very large inputs (long documents, tool outputs) push the regex and PII stages higher — budget generously if you scan multi-KB payloads.

​
ML path — hosted classifier models

When ML_INFERENCE_MODE=api, injection, toxicity, and NER-based PII delegate to hosted transformer models. Latency is dominated by the network round-trip and possible model cold-start, not local compute.
DetectorDefault modelp99 budgetNotes
Injection (ML)protectai/deberta-v3-base-prompt-injection-v250–250 ms warm503 on cold-start → auto-retry, then falls back to regex
Toxicity (ML)unitary/toxic-bert50–250 ms warm0.7 default threshold, 512 max tokens; regex fallback on outage
PII NERdslim/bert-base-NER50–250 ms warmAdds PERSON / LOCATION beyond pattern matching
The ML path fails open to regex. If the inference API times out, cold-starts, or is unreachable, the request still gets the deterministic verdict — you lose recall, not availability. Flip ML_INFERENCE_MODE=off to force regex-only (used for emergency cost/latency cutting). Self-hosted deployments can point DETECTION_ML_BASE_URL at a local TGI/vLLM endpoint to keep this path in-network.

​
Slow path — LLM-judge detectors

These call a generative LLM to reason about context. Each is opt-in per project or plan tier and guarded by a hard timeout; on timeout the detector falls open and the request proceeds on the fast/ML verdict. Treat the timeout as the p99 ceiling — the budget is “as fast as the model answers, never longer than this.”
DetectorDefault modelTimeout (p99 ceiling)Env override
LLM Guard (custom NL rules)Qwen/Qwen2.5-7B-Instruct8 sLLM_GUARD_TIMEOUT
Agentic evaluatoribm-granite/granite-guardian-3.3-8b (or meta-llama/Llama-Guard-3-8B)8 s—
LLM PII redactor (context-aware pass)instruct model6 sLLM_REDACTOR_TIMEOUT
Multi-turn escalationLLM judge8 s—
Hallucination / groundedness (RAG)LLM judge10 s—
Multimodal OCR extractionOCR subprocess8 s—
A non-instruct reasoning model on the LLM-guard slow path can blow the budget: it spends the token allowance “thinking” and gets truncated before emitting the verdict JSON, which then fails open. The default (Qwen2.5-7B-Instruct) is a non-thinking instruct model chosen precisely to answer within the 8 s budget. If you override LLM_GUARD_MODEL, pick an instruct model.

​
Keeping the tail small

  • Lean on the cache. Identical inputs reuse the prior verdict in < 0.1 ms. High cache-hit rates are the single biggest lever on p99.
  • Only enable the slow path where it earns its keep. LLM-judge detectors are for ambiguous, high-stakes surfaces (agentic tool calls, custom NL policies, RAG grounding) — not every endpoint.
  • Right-size ML_INFERENCE_MODE. api for recall, off for the lowest, most predictable latency, a local DETECTION_ML_BASE_URL for in-network ML.
  • Set fail_mode deliberately. open favors availability (allow on engine error); closed favors safety (block on error). Zero-trust projects should run closed and accept that a slow/unreachable detector then blocks rather than falls open.
  • Measure, don’t guess. These are budgets. Read your actual percentiles from the promptguard.detector.latency OTEL histogram or the Analytics Cookbook. Per-detector timings are also emitted in each event’s event_metadata.

​
Next steps

Analytics Cookbook

Measure your real p50/p95/p99 by detector and surface

Threat Detection

What each detector catches

Reliability

Fail-open vs fail-closed and graceful degradation

Rate Limits

Throughput and quota behavior