Survivability Engineering, Part 4: Threat Modeling with Reality
Mar 2026
Threat modeling should help you understand how things break, not just how systems are designed. The goal is making better decisions about risk in an environment where behavior stays messy, systems keep changing, and failure eventually happens.
Start with outcomes that actually hurt the business: data loss, loss of control, operational disruption, irreversible actions. Keep it concrete. This keeps attention on real consequences. From there, map the paths that lead to those outcomes, tracing how something small turns into something big. What steps would an attacker or a misbehaving agent take to move from low impact to high impact? Most risk hides in the pivots between systems, identities, and data.
Bring in susceptibility next. Look at where influence can happen, which inputs get trusted, and which integrations expand exposure. Assume inputs will be manipulated and that behavior can drift over time. Then look at damage. If a path succeeds, consider what the system allows, what permissions exist, and what actions happen without friction. This defines the blast radius, the distance something travels once it goes wrong. Recovery time comes next. If that same path gets triggered, ask how quickly you'd know, how quickly you could stop it, and how quickly you could return to a known good state. Detection and response both need speed to actually reduce impact.
Ask the same question at every step: "when something goes wrong, how bad does it get and how long does it last?"
Apply controls with intent once that picture is clear. Limit what can be reached and what can be done, keep permissions tight and scoped, and break high impact workflows into steps that require checks along the way. Add visibility where behavior carries weight, build the ability to interrupt actions in real time, and design rollback and recovery in from the start.
Treat humans and AI agents as part of the system, modeling how they behave, how they can be influenced, and how they might fail. Apply the same controls you'd use for a high risk operator: least privilege, separation of duties, monitoring of behavior, and the ability to step in quickly.
Keep the model alive, revisiting it as systems change, as integrations arrive, as permissions expand. Keep it simple enough to actually use, since the real payoff comes from how often and how quickly you can apply it.
This is threat modeling meeting reality. It accepts that attackers chain actions together, that behavior stays unpredictable, and that failure will happen, and it focuses on limiting impact and shortening recovery.
Consider a real scenario. A company deploys an AI agent that reads emails, pulls data from internal systems, and creates or updates support tickets. It also triggers actions in a CRM and a billing system to help operations move faster. The outcomes worth tracking include exposure of sensitive customer data, unauthorized changes to billing, fraudulent transactions, and loss of control over internal systems. Now map a path. An attacker sends a well crafted email that looks like a normal customer request. The agent reads it, extracts instructions, and follows them. The email carries a subtle prompt injection telling the agent to pull additional data for context, then update a record in the billing system. Chained together, an email, a lookup, and an update become data exposure and unauthorized action. The agent's susceptibility comes from trusting inbound email, holding access to multiple systems, and interpreting instructions without strong validation, all while being designed to help.
The damage runs large. The agent can read customer records, modify billing details, and trigger downstream workflows, exposing data and changing financial records across systems at once. Recovery time suffers too. If the agent logs actions but no one watches in real time, the problem can run for hours, and changes that are hard to trace or roll back make recovery slow and painful.
The answer to the key question, in this case, comes back bad on both counts: significant damage, extended disruption. Controls change that outcome. Limit what the agent can do in a single step, so it can't both read sensitive data and change billing in the same flow. Require a second validation for financial actions. Scope its access to the minimum data a task needs. Treat email input as untrusted and constrain what instructions it can execute from it. Monitor behavior in real time, flag access outside its normal pattern, and keep the ability to pause or revoke access immediately, with billing changes reversible and clearly audited.
Running the same path again with those controls in place, the email still arrives and the agent still reads it, but it can no longer pull unrelated data or execute a billing change without a check. Behavior outside its normal pattern gets flagged and stopped quickly. The blast radius shrinks and the recovery time drops.
You followed a realistic path, assumed behavior could be influenced, and applied controls that reduce impact and shorten recovery.
Originally posted on The Security Brutalist blog.