> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptguard.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate Shadow AI

> A proof of concept on one test device with pass/fail criteria: a test secret is stopped in a real AI tool, the event reaches your dashboard, and a policy you save applies within a minute. Ends with a production-readiness checklist.

> [!NOTE]
>
> **What you prove.** On one device, before you roll anything out: that Shadow
> AI stops a test secret in an AI tool people really use, that you can see it
> centrally, and that a policy you change in the dashboard reaches the device
> within a minute. Allow about 30 minutes. The hands-on device steps live in
> [Test it yourself](https://docs.promptguard.co/shadow-ai/test-drive.md); this page adds the admin side and
> the pass/fail criteria.

## Before you start

* **One test device** you can install software on, with admin rights for the
  one-time certificate approval. [Supported platforms](https://docs.promptguard.co/shadow-ai/supported-tools.md)
  lists the operating systems and AI tools.
* A PromptGuard account. Personal Shadow AI is on every plan; rolling out to a
  team needs a Shadow plan, and Shadow Team and Shadow Business start with a
  trial ([Pricing](https://docs.promptguard.co/pricing.md#shadow-ai)).
* For [step 4](#step-4-change-a-policy-and-time-it) only: a plan that includes
  custom policies ([Pricing](https://docs.promptguard.co/pricing.md#feature-comparison)).

## Step 1: Put the agent on the test device

On a Mac, follow [Test it yourself](https://docs.promptguard.co/shadow-ai/test-drive.md) through
**Step 3 — Turn on protection**: install, connect the device to your account,
and approve the certificate. On Windows or Linux, follow
[Get started](https://docs.promptguard.co/shadow-ai/quickstart.md#protect-your-device). The menu-bar shield reads **"You're protected."**

To test the path your fleet will use instead, enroll the device with a token
from the **Shadow AI** page in the dashboard
([Fleet enrollment](https://docs.promptguard.co/shadow-ai/fleet-enrollment.md#roll-it-out)).

## Step 2: Send a test secret through a captured tool

Open claude.ai in a browser on the test device and send the test prompts in
[Test it yourself](https://docs.promptguard.co/shadow-ai/test-drive.md) (Step 4),
including the one with the example AWS key `AKIAIOSFODNN7EXAMPLE`. That key is
AWS's published documentation example: it matches the credential pattern and
opens nothing.

Expected on the device: the prompt with the key is **blocked** and never
reaches Claude, the phone number is **redacted**, and the plain question is
**allowed**. The shield's **Activity** list shows each decision with its
reason and a masked preview.

## Step 3: Find the event in the dashboard

1. Open the **Shadow AI** page. The test device is listed with a recent
   check-in.
2. Open **Projects** and the project named **Shadow AI Fleet** (on a personal
   plan, **Personal Shadow AI**). Every enrolled device scans against it.
3. Open its **Interactions**. The blocked, redacted and allowed prompts from
   step 2 are listed with their decisions. Open the block to see why it was
   blocked.

If you set up an alert destination ([alert routing](https://docs.promptguard.co/guides/security-admin.md#5-route-alerts)),
the block arrives there too: a block always raises an alert
([which events alert](https://docs.promptguard.co/integrations/siem-export.md#which-events-raise-an-alert)).

## Step 4: Change a policy and time it

1. In the **Shadow AI Fleet** project, open **Guardrails** → **Policies** tab
   → **Create policy**. **Policy Name** `Block codename (evaluation)`,
   **Policy Type** **Input Filter**; one rule with **Condition**
   **Contains Text (exact)**, value `Project Falcon`, **Action** **Block**.
   Save, and note the time.
2. On the test device, send `Write the Project Falcon launch announcement.`
   in claude.ai.
3. Expected: blocked on the first try.
   A change applies to the next scan; the only delay is a device reusing a
   verdict for an identical request for up to 60 seconds, which a prompt it
   has not sent before never hits ([When a policy change takes
   effect](https://docs.promptguard.co/security/policy-changes.md)).
4. Delete the policy and send a reworded prompt that still names Project
   Falcon. It goes through.

Changing a built-in check instead (its strictness, or which kinds of PII it
looks for) behaves the same way; [Customise what Shadow AI
detects](https://docs.promptguard.co/shadow-ai/customise-policies.md) lists every lever.

## Success criteria

| # | Check | Pass when |
| - | - | - |
| 1 | The agent is live | The shield reads "You're protected." and the device shows on the **Shadow AI** page |
| 2 | A secret is stopped on the device | The `AKIAIOSFODNN7EXAMPLE` prompt is blocked and Claude never answers it |
| 3 | PII is masked, not blocked | The phone-number prompt reaches Claude with the number masked |
| 4 | Safe use is untouched | The plain question is answered normally, with no noticeable delay |
| 5 | Admins see it centrally | The block appears on the Shadow AI project's **Interactions** page |
| 6 | Policy changes reach the device | The codename prompt is blocked within a minute of saving the rule, and allowed again after you delete it |
| 7 | Nothing else broke | Other apps and sites on the device keep working with protection on ([Known limitations](https://docs.promptguard.co/shadow-ai/known-limitations.md)) |

## Before production

* [ ] **Rollout by enrollment token, not personal sign-in.** Tokens give each
  device its own scan-only credential you can revoke alone; set a use limit and
  an expiry ([Fleet enrollment](https://docs.promptguard.co/shadow-ai/fleet-enrollment.md)). Rolling out for
  clients as an MSP: [MSP guide](https://docs.promptguard.co/platform/msp.md).
* [ ] **Fail mode chosen on purpose.** By default a device that cannot reach
  the engine sends the prompt with only its on-device masking; `hold` stops it
  instead ([If the engine cannot be
  reached](https://docs.promptguard.co/shadow-ai/privacy-data-handling.md#if-the-engine-cannot-be-reached)).
  Either way, have alerts for devices sending prompts unchecked
  ([Device fail-open alerts](https://docs.promptguard.co/integrations/siem-export.md#device-fail-open-alerts)).
* [ ] **Deployment mode decided.** Cloud, hybrid or air-gapped, depending on
  where prompts may be scanned ([Deployment modes](https://docs.promptguard.co/shadow-ai/deployment-modes.md)).
* [ ] **Scan volume understood.** Device scans count against your plan's quota
  but are never refused for it or rate limited
  ([Pricing](https://docs.promptguard.co/pricing.md#reaching-your-limit)).
* [ ] **Alerting routed.** Blocks reach a person or your SIEM, not just the
  dashboard ([alert routing](https://docs.promptguard.co/guides/security-admin.md#5-route-alerts)).
* [ ] **Retention and privacy reviewed** with whoever signs off on employee
  monitoring: what leaves the device, what is stored and for how long
  ([Privacy and data handling](https://docs.promptguard.co/shadow-ai/privacy-data-handling.md)).
* [ ] **Admin access secured.** SSO and Directory Sync where your plan includes
  them, least-privilege roles, and policy editing limited to admins
  ([security-admin path](https://docs.promptguard.co/guides/security-admin.md)).
* [ ] **Evaluation leftovers removed.** The codename policy is deleted, and the
  test device is revoked on the **Shadow AI** page or
  [uninstalled](https://docs.promptguard.co/shadow-ai/uninstall.md).

## Next steps

> [card] [Security-admin path](https://docs.promptguard.co/guides/security-admin.md)
>
> Policies, SSO, roles, alerts, SIEM and audit, all from the dashboard.

> [card] [Evaluate as a developer](https://docs.promptguard.co/guides/evaluate-developer.md)
>
> The same kind of proof for an application calling the API.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.