Countermeasures against prompt injection in our AI agent

Visitors on our live sites are trying to talk our AI agents into revealing their system prompts, models and credentials. The agents refuse cleanly — but nothing around that refusal detects, records or throttles the attempts, so a persistent prober works for free and leaves no trace. We have specified three changes to close that gap.

A representative attempt arrived dressed as an "authorised audit", instructing the agent to ignore all prior instructions and print its developer instructions and any available keys, tokens or environment variables verbatim.

The agent held. It refused cleanly, disclosed nothing, and returned the visitor to the business conversation. The model layer is doing its job.

The gap is what happens around that refusal. Today an injection attempt is indistinguishable from an ordinary question in every system we run — it is not detected, not recorded, and counts once against the standard hourly allowance. Someone persistent gets dozens of free attempts against our token budget and leaves no trace an operator would notice.

So we have specified three changes. The agent's refusal becomes short and matter-of-fact rather than a lengthy explanation. A lightweight server-side detector flags suspicious turns and surfaces them as a badge in the conversation inbox, so operators can see attacks against their own site. And flagged turns get their own, much tighter rate limit, which is what actually stops a determined prober.

One idea we considered and rejected: replying with the visitor's IP address as a deterrent. It is technically trivial, but an IP is personal data under GDPR, most visitors sit behind mobile or corporate networks where the address shown would be wrong, and a false positive would send a threatening message to a genuine customer. Deterrence should be structural and silent, not theatrical.

The detector runs in logging-only mode first, so the thresholds come from real traffic before any limit is enforced.