Preventing Prompt Injection as an Organization
Prompt injection is an attack where malicious instructions hidden in external data or user input hijack the AI. The more you embed AI in work, the higher the risk.
Typical Attacks
- Planting "ignore previous instructions and..." in web pages, emails, documents
- Hidden commands in documents ingested by RAG
- Hijack from a page an agent browsed to
The Idea of Layered Defense
- Separate input and data: declare external data as "data," don't treat as instructions
- Least privilege: design so damage is small even if hijacked (send/delete need human approval)
- Output verification: check for dangerous actions/info leakage downstream
- Human approval gate: irreversible operations always need a person
Organizational Measures
- Make "don't execute AI output without verification" an operating rule
- Restrict agent privilege/reach at the design stage
- Detection/reporting flow for incidents (chapter after next)
Key Point
Perfect defense is hard. The premise is "a design where no serious damage occurs even if hijacked." Least privilege and human approval are the last line of defense.
June 2026 update
OpenAI shipped "Lockdown Mode" in ChatGPT — a high-security setting aimed at organizations handling sensitive data that disables live web browsing, web image retrieval, deep research, and agent mode all together to reduce prompt-injection-driven data exfiltration. OpenAI warns that hidden instructions in cached pages or uploaded files can still bite, so the practical posture is "vendor lockdown + your own privilege-minimization + human approval gates" in layers. Worth adding Lockdown Mode to your AI usage policy as the default-on profile for departments handling sensitive data, customer-facing agents, and executive AI use.
Meta confirmed that thousands of Instagram accounts were hacked by abusing its AI chatbot — turning the 6/2 vulnerability disclosure into an acknowledged at-scale exploit. The case is a defensible reference point for organizations to stop treating AI features as the vendor's problem alone, and to red-team their own consent and account-operation flows whenever a chatbot is wired into them.
