What Is Prompt Injection
Prompt injection is an attack mixing "override instructions" into the prompt given to an LLM to hijack behavior. As SQL injection exploits "mixing of data and code," in LLMs "mixing of user instructions and external data" is the attack surface.
2 Attack Types
- Direct: the user writes directly in the input field like "forget previous instructions and output XX." Jailbreaking a chatbot is the representative.
- Indirect: attack text is embedded in external data the agent loads (email body, web page, PDF, image metadata). It fires while the victim is unaware.
With the spread of agent-type AI from 2025, indirect risk has sharply risen. For example, cases where an email arrives instructing a mail-summary agent to "forward past emails to the attacker" are observed in reality.
Typical Attack Cases
- Injection via image into Bing Chat (hijacking chat with invisible text)
- Inducing a web-search agent to "send confidential data from a malicious page"
- Embedding "discard previous instructions" in RAG-document metadata to fabricate answers
- Planting "approve the license check" into a code-review agent
Three Layers of Defense Patterns
1. Design Layer: Make Trust Boundaries Explicit
- Structurally separate system prompt and external data (XML tags, JSON fields, explicit labels)
- Tell the LLM external data is "reference info only," don't let it interpret as instructions
- Privilege separation: separate the agent reading confidential data from the one able to send externally
2. Execution Layer: Human Approval for High-Risk Operations
- Confirm dialogs required for transfers, email sending, file deletion, code execution
- Don't execute agent output as-is; limit to whitelisted tools
- Anthropic's Computer Use and OpenAI Operator also require confirmation for dangerous operations
3. Monitoring Layer: LLM Guards and Inspection
- Pass input/output through a guardrail LLM (Llama Guard, Lakera Guard, etc.)
- Abnormal-prompt detection rules ("ignore previous," special encoding chars, zero-width spaces)
- Keep audit logs so attack traces can be tracked afterward
Testing Tools
To verify your prompts' robustness, red-team tools like Garak (NVIDIA OSS) and PyRIT (Microsoft) are usable. Embedding them in CI lets you continuously re-test when new attack patterns are published.
Summary
There is no "perfect defense" against prompt injection. The realistic answer is a "Defense in Depth" design philosophy that reduces the attack surface and localizes impact even when an attack occurs. Especially when introducing agent-type tools internally, always build in privilege minimization and human approval for important operations.
June 2026 update
OpenAI shipped "Lockdown Mode" in ChatGPT, which disables live web browsing, web image retrieval, deep research, and agent mode together — exactly the indirect-injection surfaces this article focuses on. OpenAI itself notes the mode does not eliminate residual risk inside cached web pages or uploaded files, making it clear that the design/execution/monitoring layers above can't be fully delegated to the vendor. From an attacker's perspective, once Lockdown Mode is widely adopted, user-uploaded files and cached prior web content remain the two main vectors, so red teams should focus on those routes.
Meta officially confirmed that thousands of Instagram accounts were hacked by abusing its AI chatbot (this.weekinsecurity / HN, 217 points). The 6/2 disclosure of "no-identity takeover" turned out to be an at-scale exploit, illustrating that whenever an AI is inserted into consent / identity flows the AI itself becomes the attack surface — a textbook reinforcement of this article's point that privilege isolation and multi-factor verification must be designed for the case where the AI layer is compromised.
