Redundant infrastructure can help a service tolerate failure, but it cannot answer every operational question. Who declares an incident? Which changes are safe during recovery? Can someone find the right credentials and restore the right backup when the usual expert is unavailable?
Resilience depends on the design of the system and the routines around it. Those routines need practice because a document that looks clear during a quiet afternoon can leave important assumptions unresolved during an interruption.
Start with the service people depend on
Describe an important user journey from beginning to end. Include external providers, identity systems, data stores and the people needed to support it. A component can report healthy while the customer cannot complete the task. Monitoring should help the team understand that difference.
Agree on the impact the organization is trying to limit. Recovery time and acceptable data loss are business decisions as well as technical targets. Make sure the people setting those expectations understand the cost and the conditions under which the targets can be demonstrated.
Test a realistic interruption
Choose a contained scenario: an unavailable provider, an expired credential or a failed deployment. Define the scope, stop conditions and recovery plan before the exercise. Begin in a safe environment, and involve the people who would actually respond.
Follow the process as written. Record where someone needs undocumented knowledge, where access is missing and where a decision has no owner. A backup check should include restoring the data and verifying that the application can use it. A rollback check should consider changes to data and interfaces, not just the previous application version.
Separate coordination from investigation
During an incident, someone must coordinate the response while others diagnose and repair. Someone also needs to communicate impact and progress to the people affected. Google’s SRE incident-response guidance describes this separation of responsibilities and the value of practising it.
The roles do not require a large organization. In a smaller team, one person may cover more than one role, but the responsibilities should still be clear. Keep a shared timeline of actions and observations so that the next responder does not have to reconstruct the incident from private conversations.
The recovery process deserves the same attention as the deployment process.
Turn each exercise into a small operational improvement
After recovery, examine the conditions that made the incident possible and the factors that shaped the response. Avoid reducing the explanation to the last person who touched the system. Ask which safeguard, information or design change would make the same situation easier to prevent or handle.
Select a manageable set of follow-up actions, give each an owner and verify that it was effective. Fixing access to a runbook or making an alert actionable can be more useful than producing a lengthy review with no completed actions. Repeat the exercise after a meaningful change in architecture or ownership. Resilience becomes dependable through evidence, rehearsal and follow-through.