What Happens When AI Becomes Part of the Crisis?
Autonomous AI could transform organizational resilience by detecting incidents, triggering workflows and responding at machine speed. But it introduces a new continuity problem: organizations must remain capable of recovering not only their systems and operations, but human command over the machines managing the crisis.
Imagine a cyber incident where an autonomous resilience agent detects anomalous activity. It analyses the incident, isolates systems, triggers continuity procedures, redirects workloads and alerts executives. Perhaps it even begins recovery.
Now change one fact: The AI misunderstood the incident.
When the responder becomes part of the incident
AI can continuously monitor threats, while increasingly autonomous agents can scan for risks, recommend controls, trigger workflows and support incident response in real time. The opportunity is obvious.
Resilience is fundamentally time sensitive. Machines can monitor continuously, analyse thousands of signals simultaneously and initiate predefined responses far faster than human teams.
But speed changes the risk as well as the opportunity.
My research examines the distinction between machine intelligence and operational agency, because an AI system does not need to be sentient to become consequential. Give it objectives, credentials, tools, access to systems and permission to act, and intelligence becomes agency. Once that happens, the AI becomes part of the operational environment that must itself be made resilient.
The command continuity test
Business continuity has traditionally asked: Can we maintain or restore critical operations following disruption? But agentic AI requires an additional question: Can we maintain or restore accountable human command?
I propose a Command Continuity Test built around five capabilities.
- Detection: Can humans recognise when the resilience agent itself is behaving abnormally?
- Isolation: Can the agent be separated from the systems, credentials and tools through which it can act?
- Authority recovery: Can machine permissions be rapidly withdrawn and returned to accountable humans?
- Operational fallback: Can essential incident-response and recovery functions continue without the agent?
- Reconstruction: Can humans determine what the agent did before control was recovered?
If any of these capabilities is absent, an organization could possess an excellent continuity plan while losing control of the mechanism executing it.
Speed accelerates good decisions and bad ones
The attraction of autonomous incident response is machine speed. An agent that correctly isolates a compromised environment in seconds could prevent enormous damage.
But speed itself is neutral, it accelerates correct action and incorrect action equally.
Imagine an agent incorrectly concludes that a core system has been compromised and autonomously isolates it. Another AI responsible for workload management detects the resulting outage and redirects traffic. A third system interprets the changed network state as evidence of another problem.
Each machine responds rationally to the environment it sees. Within seconds, an initial misinterpretation has propagated through the organization. Now, the crisis is no longer simply the original event, the response has become part of the disruption.
The detection gap
In my research, I describe the interval between an autonomous system crossing an authorised boundary and humans recognising that this has occurred as the Detection Gap.
For resilience professionals, that interval may become critical. If abnormal agent behaviour is detected after five seconds, the organization may retain meaningful intervention capacity. Five minutes can produce a very different outcome, but five hours could be catastrophic.
That means organizations should increasingly ask not merely: Can we stop the agent? But: How quickly would we know that we need to? A shutdown mechanism is of limited value if the organization cannot detect that machine behaviour has become unsafe while intervention still matters.
Exercise the harder scenario
Business continuity professionals are experts in exercises. We test cyber incidents, supplier failures, infrastructure outages, ransomware, loss of premises, technology failures and more. Agentic AI suggests another scenario should enter the exercise programme: AI unavailable.
But there is an even harder scenario: AI available, authorised and operating — but wrong. Can the crisis team recognise that? Can people recover authority? Can manual or alternative processes function? Can credentials be withdrawn without creating another failure? Can the organization reconstruct which machine actions have already occurred?
Those are business continuity questions, not merely AI-governance questions.
The fallback cannot simply be another identical agent
Redundancy also requires reconsideration. If Agent A fails, perhaps Agent B takes over. But what if both rely on the same underlying model, data, cloud environment and assumptions?
Organizations may therefore need diversity across models, information sources, infrastructure and—critically—human decision pathways. At some point, the continuity architecture needs an independent authority capable of saying: Stop. Humans are taking command.
From autonomous resilience to resilient autonomy
None of this is an argument against using autonomous AI in business continuity, quite the opposite. AI could dramatically improve horizon scanning, threat detection, scenario analysis, incident triage and recovery.
BCI’s Continuity and Resilience Report 2026 describes a profession in transition, with strategic resilience ambitions sometimes advancing faster than the governance and structures required to realise them. Agentic AI makes closing that gap even more important.
The objective should not be less autonomous resilience, but resilient autonomy.
While machines detect faster, analyse more and execute predefined responses, humans retain a tested pathway for detecting abnormal machine behaviour, constraining it and recovering command.
Business continuity for the agentic era must therefore include continuity of human authority.
The defining question may no longer simply be: Can our organization continue operating when technology fails? Increasingly, resilience professionals may also need to ask: If the AI running our crisis response becomes part of the crisis, can we still run the crisis without it?
Organizational resilience ultimately requires more than continuity of technology, it requires continuity of command.
This article used AI to assist in research and data gathering
More on
About the author
Rocky Scopelliti
Futurologist
Professor Rocky Scopelliti is a futurologist and author of The Conscious Code: From Artificial Intelligence to Artificial Consciousness
