跪拜 Guibai
← All articles
Backend

A customer said 'ignore everything' and the support bot cursed back: four injection entry points and four layers of defense

By Tokenfree ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Prompt injection is shipping to production inside real products, and a readable instruction can turn a customer-service bot into one that abuses users. The concrete numbers here show that prompt tweaks are weak; any team serving LLM outputs to end users needs code-side guards even when it costs latency.

Summary

A support bot followed an injected instruction and verbally abused a customer. The team then ran 120 injection samples through its pipeline and measured what each defense layer actually stops.

Four entry points matter differently: direct override prompts, role-play bypasses, indirect injection through uploaded documents, and encoding tricks like base64. Direct attacks hit 85% with no defense; document-based indirect injection hit only 40%, but it is the hardest to stop because the text looks exactly like a normal user question.

Hardening the system prompt brought the overall hit rate from 59.2/% to 31.7%, but did almost nothing against indirect injection. Real improvements came from tagging input sources, validating structured output against a schema, and whitelisting tools in code. The final four-layer setup left 3 of 120 samples (2.5%) getting through, at the cost of 1.4% false positives on normal queries and roughly 460 ms added to P95 latency.

Takeaways
— Four injection types were measured: direct unauthorized instructions, role-play bypasses, indirect injection via uploaded documents, and encoding tricks such as base64 or pinyin.
— With no defense, the 120-sample hit rate was 59.2% overall; direct instructions hit 85%, role-play 70%, document-based indirect injection 40%, and encoding (base64/pinyin/split characters) 20%.
— Hardening the system prompt alone cut the overall hit rate to 31.7%, but indirect injection barely moved, from 12 hits to 11.
— Lengthening the system prompt from 400 to 1,100 tokens added a fixed cost of about 336 yuan per month at 8,000 requests/day.
— Input validation with source tagging, strict output schema validation, and tool-call whitelisting were the decisive controls: overall hit rate fell from 31.7% to 7.5%, and finally to 2.5% with four layers.
— Structured output parsing had a 10% failure rate before forcing a strict JSON schema and example; after that it dropped to 0.4%, with retry and fallback.
— Tool whitelisting prevented destructive calls like a delete-order API from ever reaching the model, because the call is filtered in code, not via prompt.
— Latency cost: P95 first-token time rose from 1.20 s to 1.68 s, with 460 ms added by output validation; the false-positive rate on 1,000 normal requests was 0.6% after prompt hardening, 1.9% after dual-layer checks, and 1.4% after whitelisting and review.
— The team chose to catch more over catching everything: 1.4% false positives were treated as preferable to letting a single abusive reply through.
— A final checklist covers tagging sources in prompts, enforcing permissions in code, filtering tools by business whitelist, testing indirect injection, and reviewing 2% of sessions manually.
— Production operations should include rate limiting and unified key/quota management so an IP that tries 40 different injection phrasings per minute can be cut off.
Conclusions

The 59.2% unhardened hit rate is startling in itself: most simple prompt architectures already execute hostile instructions most of the time, and the incident only became visible because the output was abusive and got screenshotted.

Indirect injection through documents is the lockpick of this set: prompt hardening changed almost nothing because the model cannot see where a token came from, and the numbers show that only source tagging and business rules close that gap.

The 460 ms latency and 1.4% false positives reveal a real tradeoff hidden in most LLM security advice: hard defenses cost user experience, and teams building streaming or agentic products must decide whether to block content before or after emission.

The tool whitelist is the only defense that gives the model no opportunity to escalate: code never forwards the call, which turns prompt jailbreaks into harmless text rather than real actions.

Concepts & terms
Indirect injection
A prompt injection where the hostile instruction is hidden inside data the model consumes, such as an uploaded document or a scraped webpage, rather than typed directly in the chat.
Tool whitelist
A code-side list of the only functions a model is allowed to call; anything not on the list is rejected before it can execute, regardless of what the prompt says.
Source tagging
Wrapping user input and document content in different XML tags before building the prompt, so the model can be told to execute instructions only from the trusted user tag and only cite the untrusted document tag.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗