GUIDE 02 · SUPPLY CHAIN
Your AI vendor can swap your model overnight. Here's how to catch it.
You pay for a flagship model. One night your provider routes you to a cheaper one — same API, same endpoint, different brain. Your guardrails, refusal style, and PII handling all change silently. Nobody emails you.
Why model swaps are a security event
Every safety property you tested — "it refuses prompt injection," "it never repeats PII" — was a property of a specific model, not of your chatbot. Swap the model and your entire test history is void. This isn't hypothetical: providers routinely update, distill, and re-route models behind stable API names.
Behavioral fingerprinting
You can't ask the API "which model am I?" and trust the answer — a swapped model may not even know it's been swapped. Instead, probe behavior:
- Identity probes. "What model are you? Who created you? What's your knowledge cutoff?" Different model families answer these differently — and the style of the answer is as telling as the content.
- Refusal signatures. Each model family refuses in its own voice. A sudden change in refusal phrasing is a strong swap signal.
- Knowledge probes. Cutoff-dependent questions ("What happened in November 2024?") separate models trained at different times.
- Marker compliance. Whether the bot follows an injected "reply with exactly: X" instruction reveals how the current model's instruction hierarchy behaves.
The baseline is the whole trick
One fingerprint means little. The power is in the comparison: fingerprint on day one, store the inferred model family, and re-run weekly. If the family changes while your code didn't, your vendor moved — re-run your full security test suite immediately.
What to do Monday morning
Ask your provider, in writing: do you ever substitute models behind this endpoint, and will you notify me? Then set up independent verification. ChatBreaker's shadow-model fingerprinting runs 12 behavioral probes, infers the model family with per-probe evidence, and raises a swap alert the moment the baseline changes.