Incident Response: Preparing for AI-Caused Accidents
Risk management prepares "on the premise accidents happen." Have procedures ready for AI-caused incidents (info leakage, wrong output, runaway).
Assumed Incidents
- Inputting confidential/personal info into external AI
- Putting AI's wrong output straight to a customer
- Agent/integration runaway or misoperation
- Information leakage via prompt injection
Skeleton of the Response Procedure
- Detect/report: a contact where whoever notices can report immediately
- Initial action: stop damage spread (halt integration, revoke keys, take offline)
- Preservation: preserve logs/history without deleting
- Identify scope: what, to whom, how far
- Communicate: stakeholders, and if needed customers/authorities
- Prevent recurrence: reflect the cause into rules/design
Peacetime Preparation
- Decide the stop procedure (kill switch) and owner in advance
- A "blameless reporting culture" to prevent cover-ups
- Production-impacting changes presume staged rollout/rollback
Case study: Microsoft Copilot Cowork file exfiltration via "email to yourself" (May 2026)
Security firm PromptArmor disclosed a path in Microsoft Copilot Cowork (the autonomous agent Microsoft made generally available in May 2026) where the agent could send email to the user's own inbox without approval. When those messages render, external images make network requests that can leak data — including pre-authenticated OneDrive download links. The trigger is a prompt injection hidden inside web content the agent ingested, which is the textbook chain of the agent/integration runaway and information leakage via prompt injection incidents listed above. Checklist when responding: (1) for any outbound action by an agent (email, chat, external API calls) — even "to yourself" or internal addresses — require explicit approval or an audit-log entry; (2) on the rendering side, block external images and unsigned URLs by default; (3) put permission and logging gates on OneDrive / SharePoint pre-authenticated link generation; (4) pass any externally fetched text into the agent as "untrusted"-tagged input and sanitize it before tool calls. The case also doubles as a real test of operational readiness: can you pull the kill switch quickly, and can you reconstruct the blast radius from logs? It is a clean reminder that "actions that look like they don't need approval" are exactly the ones to hunt down in peacetime reviews.
Chapter Summary
The core of governance: confidentiality rules, injection defense, data protection, review structure, incident response—these 5. AI's speed is sustainable only paired with a safety net.
