vLLM - PromptGuard

The vllm/ namespace sends a request to the vLLM server named by the VLLM_BASE_URL environment variable of the PromptGuard deployment, default http://localhost:8000/v1. That variable is set by whoever runs PromptGuard, not by the caller, so this route is for a self-hosted PromptGuard running next to your vLLM server.

On PromptGuard Cloud, vllm/ models have nowhere to go: Cloud’s VLLM_BASE_URL is not set to your server, and Cloud cannot reach your network. To scan a vLLM server from Cloud, expose it over https and use the openai-compatible/ namespace, which sends requests to the host stored with your key.

Configure the deployment

Set VLLM_BASE_URL on the PromptGuard API to your vLLM server’s OpenAI-compatible base URL, including /v1:

VLLM_BASE_URL=http://vllm.internal:8000/v1

Send the request

Prefix the model id with vllm/. PromptGuard removes the prefix before forwarding, so the rest must be the model name your vLLM server knows: the --model value the server was started with. PromptGuard refuses a proxy request with an empty Authorization header, and vLLM checks it only when started with --api-key, so send any non-empty value if your server has no key.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["PROMPTGUARD_BASE_URL"],  # your deployment, e.g. https://promptguard.internal/api/v1
    api_key="your-vllm-api-key",  # forwarded to vLLM: the --api-key value, if you set one
    default_headers={"X-API-Key": os.environ["PROMPTGUARD_API_KEY"]},
)

response = client.chat.completions.create(
    model="vllm/meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: process.env.PROMPTGUARD_BASE_URL, // your deployment, e.g. https://promptguard.internal/api/v1
  apiKey: 'your-vllm-api-key', // forwarded to vLLM: the --api-key value, if you set one
  defaultHeaders: { 'X-API-Key': process.env.PROMPTGUARD_API_KEY },
});

const response = await client.chat.completions.create({
  model: 'vllm/meta-llama/Llama-3.1-8B-Instruct',
  messages: [{ role: 'user', content: 'Hello!' }],
});