Resilience is usually asserted rather than demonstrated. The architecture diagram shows redundancy, the runbook describes failover, the backup job reports success - and none of that is evidence, because the paths have never been exercised. The purpose of deliberately breaking things is to convert assumptions into knowledge at a moment of your choosing.
Start smaller than chaos engineering literature suggests, and in a non-production environment. Kill a service instance and watch whether traffic reroutes cleanly. Stop the cache and see whether the application degrades or falls over. Block a third-party API and check that the timeout and fallback behave as designed. Each of these takes minutes and each one has a decent chance of revealing something untrue in your mental model.
The restore rehearsal is the highest-value drill available and the one least often performed. Take a real backup, restore it into a fresh environment, and time the whole process with the people who would actually do it. Teams discover the same categories of problem every time: a missing encryption key, a retention window shorter than believed, a schema the current application no longer understands, and a runtime several times longer than the recovery objective they had committed to in a contract.
Run the exercises as a drill with a scenario rather than as a technical test. Someone declares the incident, the on-call rotation responds using only the runbooks, and an observer notes every point where they had to ask someone rather than look something up. Those points are the documentation gaps, and they are only visible under the conditions you have simulated - reading a runbook calmly at your desk never reveals that it omits which dashboard to open.
Have a stop condition and announce the window. This is the difference between a controlled experiment and an outage you caused: everyone knows it is happening, someone is watching real user metrics, and there is an agreed threshold at which you abort and restore. Running failure injection in production without those is not rigour, it is gambling with your customers' afternoon.
The output that matters is a short list of fixes with owners, not a report saying the system survived. In practice the first few drills find plenty - alerts that did not fire, a dependency nobody documented, a failover that requires a manual step nobody remembered. Fixing those is the entire return on the exercise, and running drills without acting on the findings is a costly form of theatre.
