Nested overlays that each capture focus can deadlock

What it is

One overlay that captures focus and still has a door is usually right. When two or more layers each claim “focus is mine”, the door hits empty space or the wrong layer: a nested capture deadlock. Typical scene: a dialog opens a date picker or a confirm sheet; Escape closes the layer underneath and leaves an ownerless top; or both layers listen for Escape and one key tears the whole stack down; or the top closes and restores focus to a trigger that no longer exists, leaving the keyboard in mid-air.

This is not “whether to capture”. It is a broken ownership stack: each layer believes it is the only captor.

Why it happens

Single-layer capture is often implemented globally: intercept Tab on document and yank focus back to the ends of “the current dialog”. When a second layer opens and registers the same intercept without pausing the first, two pieces of logic fight over one key. Escape is worse — both layers bind “close myself”; bubbling closes two layers at once, or the inner stopPropagation means the outer never hears close again.

Restoring focus depends on a pointer to “who opened me”. Nested, that pointer must be a stack: close the top, restore to the control that opened it (still inside the next layer); close that layer, restore to the original trigger on the page. A single global variable overwritten means closing the top sends focus to a page button already under the mask, or already unmounted, and the person is left in a gap that is neither top nor bottom.

Studying it

Build a minimal stack: page button → dialog → picker or second confirm inside the dialog. For each layer record: which layer received focus on open, which layer Escape closed, where focus sat after close. Operational definition of deadlock: a key sequence after which focus is on no operable control in any layer, and cannot be recovered from the keyboard alone.

Third-party widgets that open their own layer inside a first-party dialog (colour picker, payment, OAuth) are a high-rate mix and must be in the sample. Do not only test an in-house dialog wrapping an in-house dropdown.

Where it stops holding

A menu opening a submenu should not each run a modal capture. They should share one roam: arrows in and out of the submenu, Escape closing the submenu then the parent. That is a stack with a convention, not deadlock. Scenes that truly need two modals (unsaved-changes confirm over an edit dialog) can implement the stack correctly rather than banning nesting. System prompts (OS permission sheets, the browser file dialog) sit outside the page’s capture stack; the page must still reclaim focus when they close, and must not assume it is still topmost.

Applying it

• Treat open overlays as a stack: only the top captures; Escape closes only the top; on close, restore focus to the trigger stored for the next layer down.
• Do not split into two modals what can be one. When nesting is required, closing the inner layer must not dismantle the outer with it.
• How to check: open a second layer from inside a dialog and press Escape once. If both layers vanish, or focus lands on the page behind the mask, the ownership stack is already deadlocked.

Related

• Same group: J3.02.1 Focus order must match visual order · J3.02.2 Overlays must capture focus and remain escapable · J3.02.3 An inescapable focus trap is a blocking defect · J3.02.4 Dynamically inserted content desynchronizes focus unless it is moved on purpose · J3.02.5 Focus order must update as content expands or hides · J3.02.6 Focus-order failures are found by walking with a keyboard, not by inspecting source order
• Nearby: E4.10 Modal dialogs · E4.12 Popovers · J3.01 Keyboard access
• Search terms: nested modal · focus stack · stacked dialog