Survivability Engineering, Part 5: Chaos and Red Teaming
Mar 2026
Survivability engineering has so far focused on how we think about systems under attack. We assume compromise is possible, map realistic attack paths, and estimate damage and recovery. That work creates clarity, but it depends on one fragile input, our own understanding of the system, and that understanding stays incomplete.
Modern systems evolve constantly. Dependencies shift, configurations drift, controls degrade, and people adapt in ways no model fully captures, so what we believe about a system and how it actually behaves drift apart over time. Risk hides in that gap.
Closing it takes evidence, not belief.
Security Chaos Engineering generates that evidence by treating security as an experimental discipline. Teams test assumptions against reality instead of trusting that controls behave as expected, introducing controlled stress and observing what happens. Experiments expose how systems respond to failure, how controls interact under pressure, and where hidden dependencies produce unexpected outcomes.
This work stays structured and deliberate rather than random. Teams define a steady state, form a hypothesis about system behavior, introduce a specific condition, and observe the result. The outcome confirms the assumption or forces a correction, building a more accurate picture of how the system actually behaves.
The findings tend to surface fast. Detection often takes longer than expected. Containment breaks across boundaries that looked solid on paper. Recovery procedures reveal hidden coupling and manual steps that stretch out downtime. These gaps rarely show up in design documents or control matrices, since they only appear under pressure, and chaos engineering makes them visible on the team's own terms, before an adversary forces the issue.
Red teaming extends the same idea across the whole system. Done well, it works as adversarial simulation rather than a search for isolated weaknesses. The goal is understanding system behavior after access occurs, not simply proving access is possible.
A capable red team applies pressure over time, moving through identity paths, interacting with detection systems, and forcing response processes to engage. They create situations where defenders make decisions with incomplete information and real time constraints, which reveals how the organization actually operates during an incident.
The most useful discoveries are rarely technical. Teams find delays in escalation, confusion in ownership, gaps in coordination, and friction in recovery, and these factors shape how long disruption lasts and how much damage accumulates more than any individual control.
Deception adds another layer. Carefully placed signals and artifacts force interaction and generate visibility. Decoy credentials, instrumented assets, and controlled exposure points turn attacker behavior into observable data, increasing detection fidelity and reducing uncertainty during an incident while shifting some initiative back to the defender.
Chaos engineering and adversarial simulation work best together. Chaos experiments isolate and validate specific assumptions about controls and system behavior, while red teaming applies continuous pressure across the system to test how those components operate as a whole. One refines understanding at a granular level, and the other validates survivability at an operational level, both built on the same principle that systems fail according to reality rather than design.
A practical approach starts with a clear survivability question. Identify a critical function and a realistic attack path that could affect it, define what success looks like in terms of detection, containment, and recovery, then test it.
Introduce a controlled condition that reflects the attack path and observe how the system responds. Measure how long detection takes, how far the impact spreads, and how quickly recovery actions begin. Document the gaps between expectation and outcome, improve the system, and repeat.
Over time, this builds a feedback loop grounded in evidence. Threat models grow more accurate because they reflect observed behavior, recovery timelines become credible because they come from real execution, and control effectiveness becomes measurable because it has been tested under stress.
This changes how security programs evolve. Investment shifts toward reducing real attack paths, limiting actual blast radius, and shortening proven recovery time, and work that fails to improve those outcomes becomes harder to justify.
Survivability engineering reaches maturity when teams stop debating what might happen and start observing what does happen. Chaos engineering and adversarial simulation provide the mechanism for that shift, replacing assumption with evidence and turning security into an operational discipline.
If it has not been tested under stress, it remains an assumption.
Originally posted on The Security Brutalist blog.