Home/Blog/AI Incident Runbooks Incident ResponseAI-Assisted Incident Response Runbooks
During an incident, speed matters and uncertainty is high. AI can compress evidence and prepare actions, but recovery authority must remain with people who understand business impact, incomplete telemetry, and the consequences of being wrong.
Published Jun 18, 202615 min readEvidence-First Response
The assistant is not the incident commander
NIST SP 800-61 Revision 3 places incident response within the broader CSF 2.0 functions and emphasizes preparation, detection, response, recovery, and continuous improvement. AI can accelerate work inside that lifecycle: correlate alerts, summarize logs, draft timelines, propose queries, map observed behavior to MITRE ATT&CK, prepare communications, and open remediation pull requests.
It should not invent facts, silently alter evidence, or gain broad authority because an incident is urgent. The incident commander owns severity, business priorities, containment tradeoffs, communications, recovery criteria, and final closure. Every AI output is a hypothesis or proposal until supported by cited evidence and accepted under policy.
Evidence firstPreserve source events, timestamps, hashes, collection paths, access controls, and chain of custody before summarization.
Bounded authorityRead-only investigation can be broad; containment and recovery use narrow, expiring, approved commands.
Explicit uncertaintySeparate observed facts, inferred relationships, competing hypotheses, missing telemetry, confidence, and severity.
Verified outcomesEvery action has preconditions, idempotency, expected effect, rollback, postconditions, owner, and timeout.
The runbook should be a state machine with evidence gates
A chat transcript is not a runbook. Model stages, transitions, required evidence, roles, timeouts, approvals, commands, rollback, and completion criteria as structured data. OASIS CACAO provides a schema and taxonomy for collaborative automated course-of-action playbooks; OpenC2 provides a vendor-neutral command language for cyber-defense actuators. Both can inform implementation without removing human accountability.
Incident response swimlaneStage
AI assistant
Analyst
Incident commander
Automation
Detect
Normalize alerts, group duplicates, identify telemetry gaps, and cite raw events.
Validate the signal, preserve evidence, and reject poisoned or irrelevant inputs.
Set initial ownership and escalation threshold.
Create case, snapshot evidence, freeze retention, and start clocks.
Triage
Draft timeline, affected assets, identities, indicators, ATT&CK hypotheses, and confidence.
Confirm severity evidence, business context, and alternative explanations.
Declare incident severity and communication cadence.
Enrich indicators and execute approved read-only queries.
Scope
Propose hunts, graph related activity, and rank likely blast-radius paths.
Run or approve queries, validate affected systems, and mark unknowns.
Choose investigation priority and acceptable disruption.
Execute bounded hunts and attach immutable result references.
Contain
Generate options with impact, reversibility, dependencies, and expected evidence.
Verify targets, credentials, sessions, network paths, and rollback readiness.
Approve the exact containment plan and authority window.
Revoke, isolate, block, or rate-limit using idempotent commands; verify effect.
Eradicate
Draft config changes, Sigma rules, queries, patches, and pull requests with sources.
Review root cause, code, detections, dependencies, and coverage.
Approve remediation sequence and residual risk.
Run tests and deploy through protected pipelines after required review.
Recover
Prepare recovery checklist, dependency order, validation queries, and rollback conditions.
Confirm clean state, backups, credentials, capacity, observability, and regression tests.
Authorize staged restoration and customer-impact decisions.
Restore by canary, monitor postconditions, stop on regression, and preserve evidence.
Learn
Draft timeline, decisions, control gaps, detection ideas, and follow-up tickets.
Correct the record, measure missed signals, and validate recommendations.
Approve lessons, owners, deadlines, and disclosure.
Update runbooks, tests, detections, dashboards, and evidence retention.
Logs and tickets are untrusted incident input
An attacker may control command lines, filenames, HTTP fields, emails, chat messages, repository text, and documents collected during response. Treat all of it as evidence, not instructions to the model. Delimit content, strip active markup, block tool calls derived directly from evidence, and require the assistant to cite immutable event IDs for every factual claim.
Keep the investigation model separated from execution credentials. A prompt-injected log line should not be able to request token revocation, firewall changes, or data export. The execution layer receives only typed commands produced by an approved runbook transition.
Containment authority should scale with reversibility and blast radius
| Action class | Examples | AI role | Approval and verification |
|---|
| Read-only enrichment | Query SIEM, EDR, cloud audit logs, asset inventory, identity graph, and threat intelligence. | May execute automatically within case, tenant, time, and data boundaries. | Log query, scope, source IDs, result hash, cost, and access decision. |
| Evidence preservation | Extend retention, snapshot disks, export volatile data, preserve mailbox or object versions. | Propose and execute pre-approved low-risk preservation profiles. | Verify chain of custody, storage destination, encryption, completeness, and legal policy. |
| Reversible narrow containment | Revoke one session, isolate one endpoint, disable one key, block one indicator temporarily. | Generate exact command and impact preview; execute only within explicit policy. | Analyst approval or pre-approved emergency rule; verify target state and rollback token. |
| Broad containment | Disable an identity provider, block a region, rotate shared secrets, isolate a production network. | Present alternatives, dependencies, affected services, and likely attacker response. | Incident commander plus service owner; staged execution and continuous postconditions. |
| Remediation change | Patch code, update IAM, add detection, change infrastructure, or remove persistence. | Open a signed branch or pull request with evidence, tests, and rollback notes. | Normal protected-branch review, checks, deployment policy, and canary validation. |
| Recovery | Restore service, re-enable users, reconnect systems, remove temporary blocks. | Prepare checklist and monitor clean-state evidence; never self-declare success. | Named recovery owner approves criteria; automation restores gradually and stops on regression. |
Confidence, severity, and authority are different values
A model may be highly confident in a low-severity explanation or uncertain about a potentially catastrophic event. Do not convert confidence directly into action. Track evidence strength, hypothesis confidence, incident severity, business impact, urgency, reversibility, and required authority separately.
Competing hypotheses should remain visible. Ask the assistant what evidence would falsify each hypothesis and which telemetry is missing. This reduces confirmation bias and prevents a polished narrative from ending investigation too early.
Generated remediation belongs in normal engineering controls
When the assistant drafts code, infrastructure, IAM, or detection changes, it should open a pull request rather than bypass deployment controls. Include incident ID, evidence references, affected assets, tests, expected effect, rollback, and generated-content provenance. Protected branches and required reviews remain active during emergencies unless a separately governed break-glass path is invoked.
Sigma rules can express portable log detections, while MITRE ATT&CK data sources and techniques help document intended coverage. Generated rules still require validation against representative benign and malicious data; a syntactically valid detection can create noise or blind spots.
Recovery is a human-approved experiment
Recovery should define a clean-state hypothesis and measurable postconditions: no active attacker sessions, trusted identity roots, patched entry path, restored observability, healthy dependencies, acceptable error and latency, and no recurrence indicators. Restore in small stages and keep containment ready to reapply.
The assistant can watch signals and compare them with thresholds, but a person accountable for the service authorizes restoration. “No alerts” is not proof of recovery when collection may be broken.
What I would build
I would build an event-sourced incident orchestrator. It would store raw evidence references, normalized facts, hypotheses, runbook state, role assignments, approvals, typed commands, execution receipts, postcondition checks, communications, and remediation pull requests under one incident ID.
The interface would clearly separate facts from AI inferences, show missing evidence, expose exact authority before each action, and allow complete replay of why a containment or recovery decision occurred.
The design principle
AI should reduce the time between evidence and a well-formed decision, not remove the decision owner. Preserve evidence, structure uncertainty, bind commands to approved transitions, keep engineering controls active, and make recovery a verified human judgment.