Fault tolerance — the system stays recoverable after an error occurs

What it is

Fault tolerance means that once an error has already happened, the system and the user can still return to a usable, acceptable state, instead of the error simply ending the task or causing an unrecoverable loss. This is a different phase of strategy from error-proofing: error-proofing acts before the error, aiming to keep it from happening at all; fault tolerance acts after the error, starting from the premise that some errors will get through no matter what, which shifts the design question from "how do we block it" to "what happens once we couldn't." Even a system with a very high interception rate still needs fault tolerance to catch whatever slips through.

Why it happens

Fault tolerance is necessary because no error-proofing measure can drive the error rate to zero — user intent shifts, the environment throws up surprises, and the system itself will hit edge states its designers never anticipated. Pouring every resource into prevention hits diminishing returns quickly, and when an error that wasn't caught does occur, a user facing a system with no way back pays a much higher price. The core mechanism of fault tolerance is preserving the possibility of rolling back ahead of time: before an error can even happen, enough state is already saved — action history, intermediate versions, a recoverable snapshot — so that when the error does occur, the system doesn't need to improvise a fix, it just invokes a recovery path that was already prepared.

Studying it

Fault tolerance is typically evaluated through fault injection: deliberately introducing a class of error or abnormal input and observing how long it takes, and how many steps are needed, for the system and user to return to a usable state, and whether anything that shouldn't have been lost gets lost along the way. This differs in focus from pure prevention research — prevention asks "can the probability of this error be reduced," while fault-tolerance research asks "once this error happens, how expensive is recovery" — and the two lines of work track different dependent variables; fault-tolerance research cares about recovery time, how complete the recovery is, and the cognitive and operational cost the user pays during recovery.

Where it stops holding

Fault tolerance can't cover every kind of error: once an action's effect has already propagated beyond the system's boundary — a message already delivered, funds already transferred, an irreversible change already made in the physical world — no state kept inside the system can pull that external change back. In this situation, all fault tolerance can do is inform the user as fast as possible and offer a remedy, not a real undo. Whether an error is fault-tolerable depends on whether its effect has crossed the boundary of what the system can control, not on how severe the error itself is.

Applying it

When designing any flow that lets a user act, start from the assumption that an error will happen, and ask the reverse question: if this step goes wrong, does the system currently retain enough information to support recovery? Common techniques include automatically generating a recoverable snapshot or version before a critical action, logging enough detail to diagnose what happened, and pushing irreversible external effects as far toward the end of the flow as possible to lengthen the internally-recoverable window. Verification: deliberately inject one representative error into each critical flow in a test environment, and time and count the steps a user needs to get back to normal. If one flow's recovery takes noticeably more steps than others, or data is lost during recovery, that flow's fault tolerance hasn't kept pace.

Related

• Same group: A10.08.2 undo before confirmation · A10.08.4 preserving completed work under partial failure · A10.08.6 a checklist for identifying irreversible actions
• Nearby: A10.06 error-proofing design · A10.07 the Swiss cheese model
• Search terms: fault tolerance · graceful recovery · resilience