The proxy accepts
stream: true and answers with Server-Sent Events, but it does not pass tokens through as the provider generates them. It reads the provider’s whole stream, scans the completed text, and only then sends it to you. Time to first token is therefore the time to the full completion, plus the scan.How Streaming Works
- Your request is sent to PromptGuard
- PromptGuard scans the input. A blocked input gets an HTTP 403 before anything is forwarded, exactly as without streaming
- If the input passes, the request is forwarded to the LLM provider with
stream: true - PromptGuard reads the provider’s stream to the end and holds it
- In enforce mode, the completed text is scanned as output
- If the output passes, the held SSE events are sent to you in one burst. If it is blocked, you receive a single error event and
[DONE], and none of the completion
Using the OpenAI SDK
Your existing streaming code works unchanged: point the client at the proxy and add theX-API-Key header. Your provider key stays the client’s API key.
Using the PromptGuard SDK
The PythonPromptGuard client streams: stream=True returns an iterator of parsed SSE chunks as dicts. It sends only your PromptGuard key, so the request reaches the provider only if you have stored a provider key for it.
PromptGuard client does not stream: stream: true throws a PromptGuardError with code streaming_not_supported. In Node, stream with the OpenAI SDK pointed at the proxy, as above.
Server-Sent Events (SSE)
When streaming, the API returns Server-Sent Events. Each event contains a JSON chunk:Framework Integration
FastAPI (Python)
Express (Node.js)
Next.js (React)
Error Handling During Streaming
A streamed request can be blocked at two points, and each arrives differently:
The OpenAI SDKs raise both as
APIError, so one handler covers them:
Streaming with auto-instrumentation
With auto-instrumentation (init()), your client talks to the provider directly and only the input is scanned for a streamed call. scan_responses / scanResponses does not apply to streams: the Node.js SDK skips the output scan for streaming calls on purpose, and the Python SDK finds no completion text on a stream object to scan.
To scan streamed output in your own code, collect the text as it arrives and scan it with the Guard API once the stream ends:
Performance
Streaming through the proxy therefore does not lower perceived latency the way it does against the provider directly: your user waits for the whole answer either way. Keep
stream: true if your client code already handles SSE, since nothing else changes, but do not choose it for a faster first token. The scans’ own cost depends on how often your traffic escalates past the deterministic layer; measure it with the latency budgets guide.
Next steps
Streaming protection
Three ways to protect a streamed response
Latency budgets
Per-detector budgets and how to keep the guard call fast
Error handling
Every error code the API returns and how to handle it