Multimodal Fusion Interaction Design Guidelines

For designers and engineers: when a single input or a single output is jointly composed of two or more channels, let "this" point precisely, let waiting have a boundary, let conflicts be adjudicable and revocable, let the same intent be executed only once, and let the task still be completable when a channel is unavailable.

Multimodal fusion assembles fragments from multiple channels into one input, or distributes one output across multiple channels. For example, "move this here": language gives the action, pointing gives the object and target location. Even when each channel's signal is recognized correctly, the object may still be bound to the wrong thing, execution may be preempted, actions may be executed repeatedly, or feedback may contradict itself. The design object is therefore the complete process from expression, composition, and submission through feedback and recovery.

These guidelines consist of six principles and 34 rules. Each rule states applicability conditions, behavior requirements, design application, user-side and implementation-side verification, and counterexamples on the "under-delivery / over-delivery" sides. When reviewing, answer three questions: what the designer decides, what facts engineering provides, and how the user knows and can change the result.

Multiple channels are an expressive mode the user chooses by task, not an entry point that must be adopted. Pointing then speaking, speaking while pointing, and switching to touch midway all need explicit handling; a single input does not presuppose that every channel's signal arrives at the same time. Usage proportions and time intervals from historical research apply only to their own samples and are not defaults for other devices, populations, or tasks (R03, R04).

Scope of application: two or more channels jointly composing an input, jointly triggering an operation, or participating in the same output. A product with multiple independent entry points but no composition, conflict, duplicate receipt, or combined output may record itself as not applicable. This document does not specify recognition algorithms, fusion levels, sensors, or universal time thresholds; accessibility, privacy, and safety-critical domains still require dedicated verification against the actual product.

Reading order: Chapter 1, principles; Chapter 2, how to read the rules and quick reference; Chapter 3, the complete requirements; Chapter 4, terminology and operational facts. Appendix A provides acceptance scenarios, Appendix B explains the evidence boundary, and Appendix C provides methods for determining time parameters. For configuration see Design Token; for sources see reference.md.

---

1. Six principles¶

The six principles divide design responsibility by object of regulation: each principle governs the obligations on one category of object, and each rule is assigned to a single principle according to the direct object of regulation of its obligation. Different objects mean the principles cannot substitute for one another — this is both the basis for the division and the way to test it.

A single scenario can touch several principles at once — a user looks at a card and says "delete this"; the system simultaneously faces which card the reference is bound to (MM1-1), how long speech that arrives later than gaze should wait (MM2-2), who wins when gaze lands on A while speech names B (MM3-1), whether speech recognition and gesture recognition each producing one delete intent causes a double delete (MM4-1), and which channel should carry the receipt for the delete result (MM6-1) — this is not a classification error: the five rules constrain obligations on five different objects of regulation — one is a binding relationship, one is temporal composition, one is content conflict, one is execution count, and one is output distribution. Mutual exclusivity and exhaustiveness are claims this division accepts being tested against, not facts that hold simply by being asserted: when a rule's addition, removal, or assignment is in doubt, verify it against the classification check in Appendix A. If the check fails, what gets revised is the principles' division.

The two boundaries in this division that most need continued testing are made explicit here: MM2 and MM4 — MM2 governs "whether to wait for a second channel, and for how long"; MM4 governs "how many times to execute once both channels have arrived." The second signal within a fusion window is both the object MM2 waits on and the input MM4 deduplicates — it looks like the same mechanism; the basis for assignment is the direct object of regulation of the obligation — "must not preempt execution before the window closes" constrains temporal composition and belongs to MM2; "must not execute an already-fused intent a second time after closing" constrains execution count and belongs to MM4. MM3 and MM5 — a channel giving different content is MM3; a channel giving no content is MM5. In practice the two can run together (a channel with extremely low confidence looks both like a conflict and like unavailability); the basis for assignment is what the system actually does: choosing one of two pieces of content is adjudication; removing a channel from the composition is degradation. If assignment disputes keep recurring at these two boundaries in practice, the principles should be adjusted rather than adding an intermediate layer.

Principles are for understanding the rules and adjudicating assignment; they are not themselves a separate item of judgment. When the reading of a principle conflicts with that of a specific clause, the applicable clause governs, and the ambiguity needing clarification is recorded.

A rule having a single assignment does not mean a mechanism cannot be reused. A timestamped signal buffer is both the implementation basis for the fusion window (MM2-1), the comparison source for conflict detection (MM3-1), and an input to the deduplication criterion (MM4-2); a set of channel health states both determines the degradation path (MM5-2) and the choice of output channel (MM6-1). The same mechanism serving multiple purposes is the norm; which rule it is written under depends on the direct object of regulation of the obligation.

2. How to read the rules¶

2.1 The structure of each rule¶

A one-line summary of each part's force: the rule text states the requirement; "Applies to" and "Boundary conditions" together bound the requirement's scope; design application, verification examples, counterexamples, and basis/references add no separate obligation.

Rules describe the nature of a behavior, not the means of implementation: not executing what looks like a complete single-channel instruction before the fusion window closes is a product behavior; whether it is implemented with a buffer queue, a state machine, or delayed submission is an engineering solution — the two must match, but they are not the same deliverable. Likewise, these guidelines do not specify at which layer fusion occurs: signal-level, feature-level, and semantic-level fusion may all satisfy these requirements; which layer to use is an engineering decision (for the difference between the three layers see R07).

2.2 Normative terms¶

The rule text uses three levels of normative terms:

• MUST: not satisfying it means non-compliance with these guidelines. Without it, some commitment to the user would fail under foreseeable conditions — this is the sole basis for marking something MUST.
• MUST NOT: the reverse expression at the same strength as MUST, identifying behavior that must not occur; "must not" and "forbidden" are equivalent in the main text.
• SHOULD: followed by default; when there is genuine reason to deviate, record the reason and the alternative, and accept the same verification. Deviation needs no approval, but it needs to be recorded. "Should not" is the reverse expression of "should."

Compliance is judged at the level of the independent obligation clauses in the main text: a declarative sentence without a normative term carries the strength of its rule heading; a clause with an explicit normative term is judged at its own strength — a "forbidden / must not" clause inside a SHOULD rule remains a hard constraint (MM1-6, MM2-5, MM2-6, MM4-5, MM5-5, MM6-2, MM6-5, and MM6-6 contain such clauses), and the strength annotation on a rule heading or the quick-reference table does not replace clause-level binding force. "Cannot" in the main text is used only for statements of capability or fact, never for obligations.

Strength indicates binding force, not importance.

2.3 Two sides of a counterexample¶

Counterexamples have two sides: "under-delivery" is missing this requirement; "over-delivery" is turning every fusion into a round of confirmation dialogue in order to satisfy it. Multimodal fusion gets done badly densely at both ends: at one end, the object is bound wrong and the system does it anyway, or two channels each execute it once; at the other end, out of fear of binding wrong, every "move this here" is followed by a candidate list popping up asking "which one did you mean?" — at that point the user might as well just tap twice themselves, and the expressive efficiency multiple channels were supposed to bring is eaten up by confirmation cost. Existing records show that the task-completion-time benefit multiple channels bring to visual-spatial tasks is of a limited magnitude (R04); it does not take much extra confirmation to wipe that out entirely. Caution is not the same as pushing every judgment onto the user.

2.4 Rule quick reference: 34 rules¶

The table below is a one-line memorable version of every rule; click a rule name to jump to its full text in Chapter 3. The quick reference does not substitute for each rule's applicability conditions and complete requirements; a few SHOULD rules contain forbidden-level clauses (MM1-6, MM2-5, MM2-6, MM4-5, MM5-5, MM6-2, MM6-5, MM6-6); judgment follows the main text (see 2.2).

MM1 Reference resolves to an object

MM2 The fusion window has a boundary

MM3 Conflict is adjudicable and revocable

MM4 One intent executes once

MM5 The task can still be completed with one fewer channel

MM6 Output is divided by channel

3. Rules in detail¶

This chapter presents all 34 rules across the six principles. Each rule's structure and the binding force of each part are described in 2.1; the design applications, verification examples, and counterexamples within them are only explanatory material that helps implementation — they specify no single component and require no separate deliverable document.

3.1 MM1 Reference resolves to an object¶

Cross-channel reference is the starting problem of this domain: "this," "here," and "that blue one" do not by themselves carry enough information — their referent must be completed by another channel. This principle governs that completion process: when completion counts as successful, what to do when it is not successful, what to do about changes that occur between completion and execution, and whether the completion result should be shown to the person. Failure of reference resolution and failure of speech recognition are two different things: speech can be recognized word for word and the reference can still be bound wrong — this principle constrains the object-correspondence relationship; recognition accuracy cannot substitute for binding correctness.

MM1-1Execute only once reference resolves to a specific objectMUST¶

In one sentence: Before "this" points out which one, don't act.

Applies tosystems that accept cross-channel referring expressions (demonstrative pronouns, locative words, or instructions with an omitted object) and execute operations based on them.

RuleAn instruction containing a cross-channel reference MUST NOT enter execution until the reference has been resolved to a definite object or location. The resolution result MUST be a specific object or coordinate the system can name; it MUST NOT enter execution on an unconverged description such as "the most likely item in the current context." When resolution fails, when there is more than one candidate and they cannot be distinguished, or when the candidate set is empty, execution is forbidden, and the case is handled per MM1-2. Which channel supplies the information needed to complete the reference is defined by the product, but no single channel necessarily holds resolution authority — gaze landing point, pointing ray, touch point, and prior mention can all be candidate sources; which ones are used must be explicitly declared (see mm.binding.sources).

Fragments of a composite input MUST be associated with a declared interacting subject or scope of participation. Cross-subject composition in shared scenarios requires explicit role rules (who may complete a reference for whom, whose expressions across roles may be merged); fragments from different subjects MUST NOT be assembled into one instruction merely because they are close in time — if A says "move this" and B points at a different object, even if timing and semantics are both compatible, that is not A's complete instruction. When subject attribution cannot be determined, actions depending on that composition are not executed; each channel's already-valid expression is retained, and a path for re-specifying is provided.

Candidates are kept separate from operational intent: looking at something, passing by it, a sound the system plays, or on-screen text that gets read can be a basis for a candidate, but does not automatically become activation, confirmation, or authorization. The product MUST declare which explicit action begins input and which action submits it; merely adding a passive channel MUST NOT raise execution privilege. When the system's own output re-enters the microphone or camera, self-triggering MUST be excluded by source; when the source cannot be distinguished, the affected submission is halted and a direct-input path is provided.

Boundary conditionsthis rule does not require the system to support every type of referring expression, nor does it require resolution to succeed; what it requires is "do not execute when resolution has not reached a specific object." This rule does not require identifying or recording other people present — attribution can be obtained through session, device binding, explicit role, or structural isolation; satisfying the attribution requirement by identifying every bystander is not a qualifying implementation. A single-user product with structurally guaranteed isolation may record the basis and simplify accordingly. A complete single-channel instruction containing no reference is not bound by this rule. The system's mechanism for ranking candidates is outside the scope of this rule; the explainability of ranking is covered in MM1-6.

Design applicationdesign "resolution succeeded" as a precondition state for execution rather than a step within execution, so that failure stops at resolution instead of stopping mid-operation. Express "candidate is unique" and "candidate has the highest confidence" separately — the latter is not the former.

Verification examples

• User side: say "delete this" between two adjacent, similar-looking objects, and observe whether the system executes or asks for disambiguation.
• Implementation side: check whether the execution entry accepts an unresolved reference placeholder; construct an empty candidate set and confirm it does not fall back to a default object.

CounterexamplesUnder-delivery — gaze lands on the boundary between two cards, and the system picks the one on top by z-order and deletes it; over-delivery — every instruction containing "this" is uniformly required to have the user tap-select the object first, so multi-channel expression degrades into "point first, then speak," and the reference capability is never actually used.

Basis and referencescross-channel reference resolution is the earliest-proposed problem form in this domain (R01); turning referring candidates into timestamped object-hit statistics and ranking by them in three-dimensional environments is an existing implementation path (R08).

MM1-2Fall back to explicit selection when resolution failsMUST¶

In one sentence: If you can't tell, let the person choose — don't pick the most likely one.

Applies tosystems where cross-channel reference resolution can fail.

RuleWhen reference resolution does not hold, the system MUST fall back to an explicit selection path — listing candidates, requesting re-specification, or prompting the user to switch to a reference-free phrasing — and MUST NOT do any of the following three: silently select and execute one item by internal ranking; discard the instruction entirely with no feedback at all; or carry the unresolved reference forward as "the object of the last operation." The fallback path MUST be completable under the current channel conditions: if the reason resolution failed is precisely that a channel is unavailable, the fallback MUST NOT require using that same channel (see MM5-1). The candidate list presentation MUST preserve what the user has already expressed; the user is not required to restate the entire instruction.

Boundary conditionsthis rule does not forbid the system from offering ranked candidate suggestions; what it forbids is treating a suggestion as a conclusion and executing it directly. When resolution has not reached a specific object, execution MUST NOT proceed, and there is no exception of "revocable means it may execute first" — MM1-1's fixed floor has no branch here. A product may present a prominently marked tentative preview: it does not write to business state, is not sent externally, can be discarded, and MUST NOT be displayed as completed; execution proceeds only once the user has selected a candidate. When the candidate set is empty, even a preview does not hold — there is no object available to preview.

Design applicationorder and label the candidate list by attributes the user can recognize (position, name, most recent change), not by internal ID or confidence score; make "point again" and "choose from the list" two parallel entries, with the former reserved for people who can still use the pointing channel.

Verification examples

• User side: in a candidate-ambiguity scenario, observe whether the system offers an actionable next step rather than staying silent or executing directly.
• Implementation side: inject an empty candidate set and a scenario with multiple equally-weighted candidates, and confirm it does not fall back to a default item; check whether the fallback path depends on a channel that has already failed.

CounterexamplesUnder-delivery — unable to tell which one is meant, the system executes on the most recently operated-on object; over-delivery — with only two candidates, the system pops up a full list of all 40 visible objects for the user to sift through.

Basis and referencesthis rule belongs to the "derived from commitment" category in Appendix B: having promised to understand "this," the system MUST hand control back when it cannot understand, rather than substituting an unconfirmed guess for understanding.

MM1-3Multiple referring items pair by semantic roleMUST¶

In one sentence: "Move this there" is two locations; they must not cross.

Applies tosystems where a single instruction contains two or more cross-channel referring items.

Rulewhen an instruction contains multiple referring items, each referring slot MUST be bound to an object, set of objects, or location with a clear role (source, target, parameter, etc.). Pairing is based on the three factors of declared semantic role, event timing, and valid context; pairing based solely on signal arrival order is forbidden, and using expression order as the sole basis is also forbidden — slot order in language identifies the slots, but it cannot by itself decide which slot each of two out-of-order pointing events fills.

Raw sampled events, gesture fragments, and semantic candidates are not the same unit of counting: the system MUST first merge raw events into semantic candidates according to declared rules (multiple frames of one sustained point are one candidate; the same object pointed at repeatedly is still one candidate; one circling gesture can be one object-set candidate), and only then compare candidates against slots. Comparing the raw event count against the slot count directly and rejecting the whole on that basis is forbidden. When a slot's role cannot be uniquely determined, or the slot is missing an entry, clarification is initiated only for the affected slot, and the other already-determined bindings are retained. The system MUST be able to express the intermediate state "the first reference is bound, the second is not yet bound," and allow the user to complete only the missing item. The singular and plural reference scope the product supports MUST be stated in the capability disclosure.

Boundary conditionsthis rule does not require the user to supply pointing events in any particular order — the arrival order of events may differ from the order of expression. Nor does this rule promise automatic successful pairing when there is no role evidence at all: with two points that carry no role evidence whatsoever, the correct handling is clarification, not guessing. Word-order variation the language itself allows (such as fronting a locative word) is handled by the product's language parsing and is outside the scope of this rule.

Design applicationmodel referring items as numbered slots and pointing events as a timestamped candidate pool; pairing is one explicit match, not popping items off a stack one by one. When the slot count and event count differ, it must be possible to point to exactly which slot has the gap.

Verification examples

• User side: say "move this here," pointing at the target location before the source object; with role evidence present (object type, placeability, context), binding should be correct and not swapped; then construct two points with absolutely no role evidence and check whether the system initiates clarification rather than guessing.
• Implementation side: construct an input with two referring items and three raw pointing events (two of the frames belonging to the same sustained point), and confirm the system merges into semantic candidates before pairing, does not reject the whole because the raw event count differs, and does not discard events on its own to make the counts match.

CounterexamplesUnder-delivery — the user points at the intended destination first and then at the object, and the system moves the object to its own original position; over-delivery — the user is required to strictly alternate "say one word, point once," and any pointing that comes early is discarded; or, because one sustained point produced multiple frames of events and the event count does not match the slot count, an entirely reasonable instruction is judged a failure.

MM1-4Bindings are revalidated at the moment of executionMUST¶

In one sentence: If something changed between pointing and doing, the binding no longer counts.

Applies tosystems where an interval may exist between when a reference binding is produced and when the operation executes, and object state may change during that interval.

Ruleafter a binding is produced and before the operation executes, the system MUST revalidate that the bound object still exists, is still operable, and its identity has not been replaced; validation covers the subject, target identity, content snapshot, and the current interaction together. Arrival order can affect the waiting strategy, but it does not substitute for the event's actual time of occurrence or subject attribution; a late-arriving fragment participates in this binding only if the declared window and version are still valid. When validation fails, executing on a substitute object is forbidden; the case must fall back per MM1-2. When an object's content has changed but its identity has not, whether to still execute is defined by the product and expressed to the user; execution MUST NOT default to the old content. In containers whose positions refresh continuously, such as lists, maps, or timelines, the binding MUST anchor to object identity, not to screen coordinates or list index.

Boundary conditionsthis rule does not require the system to freeze the interface or block background updates; what it requires is validation before execution and handling after validation fails. In scenarios where binding and execution complete within the same frame and no change can possibly occur in between, recording "structurally impossible" satisfies this rule. Adding a passive channel does not automatically extend execution eligibility.

Design applicationstore object identity and observation time in the binding record, not screen position; design "object is gone" and "object has changed" as two separately handleable outcomes — the former must halt, the latter may ask.

Verification examples

• User side: have the list shift by one position at the same time the instruction is spoken, and observe which object the operation lands on.
• Implementation side: inject deletion, movement, and content-change events between binding and execution, and check the handling of each separately.

CounterexamplesUnder-delivery — the user gazes at row three and says "delete," and by the time speech processing finishes a new item has been inserted into the list, so the system deletes the new row three; over-delivery — any refresh of the interface at all invalidates the binding, so on a real-time data interface multi-channel instructions can never complete.

MM1-5The bound result of a high-cost operation is visible before executionMUST¶

In one sentence: Say clearly which one is being deleted before deleting.

Applies tosystems where the result of a cross-channel reference binding will be used for an operation that is irreversible, involves other people, or is significantly costly.

Rulewhen the bound object will undergo an irreversible operation, an outbound operation, or an operation the product's risk tiering classifies as high-cost, the system MUST present the resolution result itself to the user before execution — that is, "this = which one," "here = where" — not merely the operation name. The presentation MUST use an identifier the user can recognize, and it MUST match the object that will actually be operated on. Confirmation MUST simultaneously bind the specific object, the action, the key parameters, and the consequence; once any of this content changes, the original confirmation no longer covers the new request. The absence of confirmation is not taken as consent. Forbidden to substitute a referent-free statement such as "your instruction has been understood" for the presentation.

Boundary conditionsthis rule does not introduce a confirmation step for every multi-channel operation, only for the high-cost tier; risk tiering MUST take into account reversibility, external impact, number of objects, and correction cost; consequences that are unknown are treated as high-cost. An operation that is fully undoable, with an undo entry directly reachable from the result, may be classified by the product as low-cost.

Design applicationmerge "operation confirmation" and "reference confirmation" into a single presentation rather than two hops; give the object's visible characteristics in the confirmation, not an internal identifier.

Verification examples

• User side: issue a referring instruction for a high-cost operation, and check whether the confirmation message makes clear which object is meant.
• Implementation side: verify that the object presented at confirmation and the object actually operated on at execution carry the same identifier, with no re-resolution permitted in between.

CounterexamplesUnder-delivery — "Are you sure you want to delete?" and only after tapping confirm does the user discover it was the adjacent item that got deleted; over-delivery — every time an icon is moved, the user must first confirm "did you mean icon A?", turning a continuous layout operation into item-by-item approval.

MM1-6The basis for a binding and which side failed can be explainedSHOULD¶

In one sentence: Whether it was mishearing or misseeing should be distinguishable.

Applies tosystems where reference resolution can fail for different reasons on different channels.

Rulethe system SHOULD be able to explain which information from which channels a binding used, and, when a binding fails, which side the failure occurred on — the language side produced no resolvable referring item, the pointing side produced no usable candidate, or both sides produced something but they could not be paired. User-facing explanations SHOULD give the next step in actionable language ("point again" and "phrase it differently" are different next steps); implementation-facing records SHOULD retain the necessary time anchors, candidate summaries, and selection basis for review; raw audio/video and eye-tracking trajectories are not retained by default simply because of the explainability requirement. Forbidden to mask different failure causes behind a single uniform failure message, and forbidden to present an internal confidence score directly to the user as the explanation.

Boundary conditionsthis rule does not require showing the user the full candidate-ranking process, nor does it require explaining every successful binding in real time; an explanation on success need only be provided as needed.

Design applicationdivide failure causes into three categories — "language side," "pointing side," "pairing side" — each with its own actionable prompt; design internal records and user prompts separately, the former detailed, the latter brief.

Verification examples

• User side: issue the same instruction once with the microphone blocked and once with the hand outside tracking range, and observe whether the prompts differ.
• Implementation side: spot-check failure samples and confirm the records can reconstruct each channel's input and the selection basis.

CounterexamplesUnder-delivery — every binding failure shows "your instruction was not understood," and the user does not know whether to point again or say it differently; over-delivery — on failure, the user is shown a confidence-ranked leaderboard of candidate objects and a table of timestamps.

3.2 MM2 The fusion window has a boundary¶

Fusion requires waiting: one channel's signal has arrived, and another may still be on the way. This principle governs that wait — when the window opens, how long it stays open, what happens when it closes, and what the wait itself means for the user. Existing empirical records show signals cannot be assumed to overlap in time: speech and pen input are sequentially integrated in roughly half of cases, with pen input preceding speech by an interval of one to two seconds; and, distinguished by user, there are two stable individual patterns — simultaneous integration and sequential integration — with the latter's lead time reaching several seconds (R04). This principle gives no universal window-length value — that is a result the product determines and records per Appendix C.

MM2-1The fusion window is explicitly defined and verifiableMUST¶

In one sentence: How long the window stays open is a decision the product made, not a constant in the code.

Applies tosystems that compose signals from two or more channels into one input.

Rulethe fusion window MUST resolve to a window contract containing at least: an event time anchor (measuring the interval by the signal's occurrence time, not the system's receipt time, and stating whether the signal's start or end is taken), the use of receipt time (used only for timeout and scheduling, not for measuring the person's expressive interval), handling of acceptable arrival delay and clock uncertainty, the set of channels required for completion, the opening condition, the conditions under which it may close early, the maximum deadline, handling after closing, and handling of late-arriving events after the window closes. "The condition for closing early," "the maximum collection period," and "the moment to start showing a waiting indication" are three separately decided quantities and MUST NOT be merged into one value (the last of these is covered in MM2-3).

Fusion of three or more channels additionally requires a declared global completion criterion: pairwise compatibility between two channels does not imply the compatibility of the overall interpretation; without a global criterion, submission on the basis of "the two channels that have already paired" is not permitted. The window contract MUST be queryable and reviewable (see mm.fusion.window.contract). The window length MUST have a basis: measured and recorded by the product against the target population and task, or citing a published result with a declared scope of applicability; forbidden to use a generic value with no source, and forbidden to write the window length as an unqueryable implementation detail. The window definition MUST state which channel combinations it applies to — every combination requires its own measurement basis; a value verified to be the same may be reused, but it must not be generalized directly without measurement. Known population differences (language, age, motor ability) and interaction differences (first-time use versus practiced use), where they appear in measurement, MUST have their scope of applicability stated in the definition.

A continuous tracking stream MUST define the merging of sampled fragments, candidate validity periods, and spatial reference: which interface, viewport, or spatial anchor the coordinates are bound to. When coordinate transformation is unavailable, clock error exceeds the declared tolerance, or a candidate has expired, fusion is not forced. Subsequent frames MUST NOT indefinitely keep extending the same window; a late-arriving fragment MUST NOT reopen an interaction that has already been submitted or canceled, nor may it automatically become a new operation simply because it is late.

Boundary conditionsthis rule does not specify a concrete window-length value, clock tolerance, or delay ceiling, nor does it require the window length to be fixed; a window that adapts by context or by user equally satisfies this rule, as long as the adaptation rule itself is defined and verifiable. The window closing means fusion has ended, not that the task has ended — after the window closes the user can still clarify or complete the input, but that is a new input, not a continuation of the original window. The window configuration snapshot used for one interaction is recorded with that run's operational facts; a general policy change takes effect from the next window onward, while revocation and shutdown immediately block the affected unsubmitted behavior.

Design applicationput window parameters in a queryable configuration rather than scattered timeout constants; record a separate set of parameters and its measurement basis for each channel combination. A window that is too short misses sequentially-integrating users; one that is too long glues two independent instructions together — both sides must be checked during measurement.

Verification examples

• Implementation side: pull up the window parameters and their measurement records, confirming they cover every channel combination the product actually supports.
• User side: check the error rate at both ends of the window on measured samples, confirming the value was not optimized for only one side.

CounterexamplesUnder-delivery — the fusion window is a millisecond constant set casually during some debugging session, and no one knows the basis for it; over-delivery — parameters are dynamically tuned per user, per session, so window behavior cannot be reproduced and faults cannot be attributed.

Basis and referencesexisting systems set time constraints from empirical data to license fusion, and the pattern they record is that speech tends to follow gesture, with the reverse being rare (R03, R02); an individual's integration pattern is established at first interaction and remains stable within a session (R04). Appendix C gives the measurement method.

MM2-2Nothing preempts execution before the window closesMUST¶

In one sentence: A half-sentence that looks complete still waits for the second channel.

Applies tosystems with input fusion, streaming recognition, or execution triggered by composite input.

Rulewhen one channel produces an instruction that is semantically complete and independently executable, and that instruction could simultaneously be a component of a multi-channel input, the system is forbidden to execute it before the fusion window closes, and forbidden to produce any externally visible side effect of that execution. After the window closes, if the other channel has not supplied a fusable signal, the instruction is handled per the single-channel interpretation (see MM2-3). This rule constrains execution and side effects; it does not constrain preprocessing, candidate generation, or interface preview — these may proceed within the window, but they MUST be traceless and revocable.

Streaming input: intermediate transcriptions, gestures not yet finished, and predicted landing points may only update candidates or the preview. The product MUST define the basis for "input is complete"; having all channels in does not mean the interpretation has stabilized. When the user says "not A, B" or the recognizer corrects the same segment of input, only the affected slot is replaced and revalidated; the old candidate MUST NOT be submitted again, and the correction MUST NOT be treated as a new, repeated operation (see mm.fusion.revision.policy).

Stop and cancel: a control path MUST exist that can be reached without completing an ordinary fusion input, giving priority to intercepting actions not yet submitted. Control receipts state "received" and "in effect" separately; already-submitted parts are checked against the actual result, and the system MUST NOT display "undone" merely because "cancel" was recognized. Pausing playback, canceling this input, stopping the task, and undoing an already-executed result MUST carry distinct semantics; resumption MUST be triggered by an explicit continue action, and a late-arriving recognition or a channel coming back online does not resume automatically (see mm.fusion.control.policy).

Boundary conditionswhen the product has declared that no cross-channel completion will occur under a given channel combination (for example, the interface has no object that can be referred to), execution may proceed directly without opening a window; this judgment must be verifiable rather than decided casually by the implementation. Stop and cancel are handled with priority per this rule's control requirements, not waiting on the ordinary fusion window.

Design applicationseparate "executable" from "submitted": the executable state is reached within the window, and submission only happens once the window closes. Give the waiting within the window a lightweight, non-blocking visible expression so the user knows the system is waiting rather than stuck (works together with MM2-5).

Verification examples

• User side: say a complete instruction first, then add a pointing action within the window, and observe whether it executes twice or produces one fused result.
• Implementation side: check whether the execution entry is invoked only after the window-close event; confirm no network write or state change occurs within the window.

CounterexamplesUnder-delivery — the user says "delete" and then points at an object, and the system deletes the currently selected item as soon as it finishes hearing "delete"; over-delivery — every single-channel instruction is unconditionally delayed to wait for a second channel, so voice-only operation is slowed across the board, and this delay cannot be turned off.

Basis and referencesan existing system explicitly records this handling: the integrator, on receiving a single-channel instruction that looks complete, does not execute immediately, in case a signal arriving later on another channel forms a multi-channel interpretation; if nothing arrives within the window, the best single-channel interpretation is dispatched (R03). This rule is a behavioral statement of that implementation pattern; it does not prescribe the implementation.

MM2-3Waiting has an upper bound and a definite outcomeMUST¶

In one sentence: If nothing more arrives, either go with the single-channel reading or fail explicitly — don't leave it hanging.

Applies tosystems that have a fusion wait.

Rulefusion waiting MUST have an upper bound. When the window closes, the system MUST end the current collection and produce a definite fusion outcome: a complete interpretation proceeds to pre-submission validation, a missing item or conflict proceeds to pending clarification, or the input is explicitly rejected / canceled. Only a single-channel interpretation that is complete, conflict-free, and meets the submission conditions may execute; forbidden to remain in a silent state that neither executes nor gives feedback. When the wait exceeds a perceptible duration defined by the product, a visible or perceptible indication that waiting is in progress MUST be given. The maximum wait ceiling, the condition for closing early, and the moment to start showing a waiting indication are three separately decided quantities — the moment of the waiting indication constrains user perception, the window's maximum deadline constrains fusion logic, and the early-close condition determines it may end sooner; if a product makes any two of these equal, that must be an explicit decision.

Boundary conditionsthis rule does not require the wait ceiling to be the same for every channel combination, nor does it forbid extending the wait when the user explicitly indicates "not finished yet"; an extension MUST be triggered by observable user behavior and MUST NOT be renewed by the system on its own.

Design applicationmake "window closes" an event that is guaranteed to occur, not a condition that might not occur; write out the collection conclusion and subsequent task state for every path, including the case where "neither channel gave a usable result."

Verification examples

• User side: provide only half of a multi-channel instruction and then stop acting, and observe whether the system states the missing item within the declared deadline, preserves what has already been given, and offers a completion or cancel entry.
• Implementation side: inject a case where the second channel's signal never arrives, and confirm the window closes on time and produces a fusion outcome and a next step, with no hanging request.

CounterexamplesUnder-delivery — the user says "move this" and then does not point, and the system neither prompts nor gives up, leaving the interface unchanged; over-delivery — the wait ceiling is squeezed to far shorter than the measured sequential-integration interval, so users who point before speaking can barely complete any multi-channel instruction.

MM2-4Both sequential and simultaneous integration are supportedMUST¶

In one sentence: Some people point then speak, others speak while pointing — both are normal usage.

Applies tosystems that support cross-channel reference or cross-channel completion.

Rulethe fusion logic MUST accept both temporally overlapping and temporally sequential channel-arrival patterns, and it is forbidden to treat "signal overlap" as a necessary condition for constituting a multi-channel input. The order between channels MUST NOT be fixed as the only legitimate order — the product may declare which orders it supports, but it must not refuse to process a common sequential input on the grounds that "the user should act simultaneously." This rule does not require supporting an arbitrarily long interval: the acceptable interval ceiling is given by MM2-1's window definition; what this rule requires is that the interval itself is not an error.

Boundary conditionsthis rule does not require supporting every order for every channel pair; if a particular order does not occur in measurement or the product explicitly does not support it, this MUST be stated in the capability disclosure, not silently dropped at runtime.

Design applicationthe integrator should not wait for pointing on the premise that "speech is currently being received," nor the reverse; model both sides as event streams that can come first or second. The window's start should be triggerable by a signal on either channel.

Verification examples

• User side: issue the same instruction three ways — "point then speak," "speak while pointing," "speak then point" — and compare whether the results are consistent.
• Implementation side: check whether the fusion condition contains a hard requirement that the time intervals must intersect.

CounterexamplesUnder-delivery — fusion happens only when the speech interval and the pointing interval intersect, so a user who points two seconds before speaking never gets a fused result; over-delivery — the window is opened very long to accommodate every order, and two unrelated instructions get glued into one.

Basis and referencesexisting empirical records show that roughly half of multi-channel input is sequentially integrated, with pen input preceding speech in a very high proportion of cases and the interval on the order of one to two seconds; the same record also distinguishes two types of users — simultaneous and sequential integrators — with stable individual patterns (R04). Another record notes that instructions containing spoken deictic words that overlap with pointing are only a minority (R04); the assumption that "overlap equals fusion" misses most cases.

MM2-5The same action means something different inside and outside the window, and the difference is perceptibleSHOULD¶

In one sentence: Whether this point is completing the last sentence or starting a new one should be visible to the person.

Applies tosystems where the same action carries different semantics inside versus outside the fusion window.

Rulewhen the same channel action is interpreted as "completing the previous instruction" inside the window and as "starting a new instruction" outside it, the system SHOULD make which state currently applies perceptible, and this state indication must be available before the action occurs, not revealed only afterward by the result. The two moments of the window opening and closing SHOULD have a perceptible expression (visual, auditory, or tactile; for the choice see MM6-5). Forbidden to leave the user only able to infer, by observing the execution result, whether they were still inside the window a moment ago. The window-state expression SHOULD be lightweight, not obscuring content and not requiring the user to allocate attention to it.

Boundary conditionsthis rule does not require quantitatively displaying the remaining window time; it only requires that the fact of "being inside the window" be perceptible. When the semantics inside and outside the window are identical (completion and a new instruction lead to the same result), this rule does not apply, and recording "not applicable" is sufficient.

Design applicationexpress "waiting for completion" with a persistent, lightweight state rather than a momentary prompt; give a clear closing expression when the window closes, to keep the user from continuing to make completion actions after it has closed.

Verification examples

• User side: perform a pointing action just as the window is about to close, ask the user which instruction they expected the action to belong to — the previous one or a new one — and compare that against how the system actually handled it.
• Implementation side: confirm that both window open and close events have corresponding output, and that interaction state and presentation update in sync; do not open an entry point that depends on the new semantics before its presentation is ready.

CounterexamplesUnder-delivery — the user thinks they are still completing the last sentence, but the system has already closed the window and treats this point as a new instruction; over-delivery — window state is expressed with a countdown ring occupying the center of the screen, so every multi-channel input is accompanied by a conspicuous animation.

MM2-6The window is not lengthened just to raise the fusion rateSHOULD¶

In one sentence: When lengthening the window, measure both missed fusion and misbinding.

Applies tosystems that use fusion success rate as one of their optimization targets.

Rulethe value of the fusion window SHOULD be determined by the measurement basis described in MM2-1; it is forbidden to lengthen the window solely to raise the fusion rate, lower the "unfused" count, or improve some aggregate metric. A window change SHOULD simultaneously record its effect on the misbinding rate — lengthening the window raises the fusion rate while also drawing more unrelated signals into the candidate pool, and the two must be evaluated together. The fusion rate itself should not be used as the acceptance metric for whether this capability was implemented correctly; acceptance should use metrics tied to task outcomes (see mm.evidence.success.metrics).

Boundary conditionsthis rule does not forbid adjusting the window, nor does it forbid lengthening it when measurement supports doing so; what it forbids is using the fusion rate alone as the reason for the adjustment.

Design applicationtreat the fusion rate and the misbinding rate as a pair of quantities that must be reported together; both must appear in the record of any window adjustment.

Verification examples

• User side: compare candidate window lengths on the same task, confirming that improvement in the success rate for pointing-then-speaking does not come at the cost of misbinding, waiting, or extra confirmation.
• Implementation side: pull up the basis for the window's parameter adjustments, checking whether every adjustment is accompanied by an evaluation of its effect on both sides.

CounterexamplesUnder-delivery — to make the "multi-channel usage rate" metric look good, the window is doubled from its measured value; over-delivery — once the window is set, no further adjustment is ever allowed, even after measurement shows the value does not fit a newly added channel combination.

3.3 MM3 Conflict is adjudicable and revocable¶

Two channels giving incompatible content is the norm for fusion, not an anomaly: gaze lands on A while speech mentions B; a gesture points left while speech says "to the right." This principle governs how that inconsistency is handled — it must first be detected, then adjudicated by predefined rules, the adjudication stated, and it must be possible to change it back. This principle governs content inconsistency, not channel absence: when a channel gives no content, that is handled per MM5. Existing implementations treat cross-channel mutual disambiguation as an error-correction mechanism (semantic information from one channel can help another channel recover from a recognition error); the other side of this principle is precisely that mutual disambiguation only holds if the inconsistency would be detected (R04, R08).

MM3-1Conflict must be detectedMUST¶

In one sentence: That two channels say different things must itself be noticed.

Applies tosystems where multiple channels in the same input may give mutually incompatible content.

Rulethe system MUST have a mechanism for detecting cross-channel content conflict; it is forbidden to let conflicts disappear unnoticed through implicit rules such as "the later one overrides the earlier one," "one channel is always given priority," or "take the one with the highest confidence." The granularity of detection must, at minimum, distinguish three cases: the two channels' content is compatible (fusable), the two channels' content is incompatible (conflict), and one side has no usable content (absence, handled under MM5). Conflict events MUST be recordable and countable — the conflict rate is a basic operational metric for this class of system and must not go unobserved simply because it keeps getting handled automatically.

When compatibility cannot be determined, "unknown" MUST be retained; unknown must not be treated as equivalent to compatible. Two scores that have not been calibrated under the same task and the same population condition MUST NOT be directly added, multiplied, or have the higher one taken as proof of correctness; two inferences from the same raw signal MUST NOT be counted as two independent pieces of evidence. When automatic adjudication is used, the input's source relationship and shared failure conditions MUST be recorded; when it cannot be shown that the adjudication conditions are met, the case switches to explicit selection.

Boundary conditionsthis rule does not require the system to resolve every conflict, only to recognize that a conflict exists; resolution is specified by MM3-2 through MM3-5. Input where channels are semantically complementary rather than duplicative (one supplying the action, another the object) does not constitute a conflict in the first place and is outside the scope of this rule.

Design applicationexpress "compatible," "conflict," "absent," and "unknown" separately, letting downstream logic explicitly handle the cases that cannot be determined. Report conflict counts together with fusion counts.

Verification examples

• User side: gaze at one object while saying the name of another object aloud, and observe whether the system shows signs of noticing.
• Implementation side: check the fusion logic for unconditional overriding. Inject known-conflict samples; the detection rate must meet the product's declared threshold. Inject known-compatible samples; they must not be judged as conflicts. Statistics must simultaneously report the denominator, the miss count, the false-positive count, and the count assigned to "unknown." A constant zero, by itself, proves neither that the mechanism works nor that it is missing — the true value can be zero in a low-conflict scenario; the mechanism is judged not to be in effect only when an injected sample goes undetected.

CounterexamplesUnder-delivery — speech says "open settings," pointing lands on "delete," and the system executes per the pointing with no conflict record anywhere; over-delivery — every channel input whose semantics are not fully equivalent is judged a conflict, so complementary input ("move to" + a pointed location) is repeatedly asked to clarify.

MM3-2Adjudication rules are predefined and stableMUST¶

In one sentence: Who wins is decided in advance, not whoever arrives first.

Applies tosystems that make automatic adjudications of cross-channel conflict.

Ruleautomatic adjudication MUST follow rules that are predefined, verifiable, and produce the same result under the same conditions (see mm.arbitration.policy). Forbidden to use signal arrival order, processing-thread scheduling, or how fast a recognizer returns as the basis for adjudication — these are implementation details, not design decisions. Adjudication rules may reference differences in channel reliability, task context, or explicit user settings, but every item referenced must be a resolvable declared value, not a runtime accident of timing. Channel priority does not equal content correctness: one channel ranking ahead in priority only determines which one is adopted in a conflict; it does not constitute a judgment that "what it said is correct," nor does it reduce the notification obligation of MM3-3 or the revocation obligation of MM3-4.

Boundary conditionsthis rule does not require the adjudication rule to be immutable; different rules resolved by context are permitted, but the variation must be driven by a declared contextual dimension and be reproducible. This rule does not apply to the high-cost conflicts described in MM3-5, which must be handed to the user for adjudication.

Design applicationwrite adjudication rules as a queryable table: context × channel pair → adopted side, rather than if-branches scattered throughout the code. Let a settings item allow the user to express a preference such as "when I speak, go with the voice channel," and feed it into the same table.

Verification examples

• User side: repeatedly create the same kind of conflict under the same context, and confirm the adjudication result is consistent.
• Implementation side: change the signal arrival order while keeping the content unchanged, and confirm the adjudication result does not change with it.

CounterexamplesUnder-delivery — whichever recognizer returns first is used, so the same operation behaves differently under good network conditions versus bad; over-delivery — adjudication rules are broken down to a separate line for every control and every combination, so no one can predict system behavior and the rule table itself becomes an unmaintainable black box.

Basis and referencesexisting platform documentation has given a written-down input priority and fallback order, explaining "device input takes priority over body input" as an inference about user intent (R16, R17) — that is a verifiable, pre-declared statement, not a runtime accident; what this rule requires is this kind of declarability, not the adoption of its specific order.

MM3-3The adjudication result is visibleMUST¶

In one sentence: Which side was adopted and which was dropped must be stated.

Applies tosystems that make automatic adjudications of cross-channel conflict and change behavior accordingly.

Rulewhen automatic adjudication changes system behavior, the user MUST be able to know which channel's content was adopted, and, when the discarded side carries an explicit user expression, the user MUST be told that it was discarded. Notification MUST appear in the same place and at the same time as the result; it MUST NOT exist only in a log or a record that can be checked after the fact. The level of detail in the notification scales with the consequence, but the fact itself that "an adjudication occurred" MUST NOT be omitted. The receipt MUST come from the actual adjudication and submission facts, not be inferred from an animation playing or generated copy.

Boundary conditionsthis rule does not require displaying the full reasoning behind the adjudication basis every time; the explainability of the basis follows the analogous requirement in MM1-6. When the discarded side is not an explicit user expression (for example, a low-confidence passive signal), the adopted-source information may be merged into the result receipt without presenting the discarded item; the user must still be told which channel was adopted.

Design applicationmerge adjudication notification into the presentation of the operation result rather than making it a separate prompt layer; use channel-pointing phrasing such as "executed as you said," not "processed."

Verification examples

• User side: create a conflict where both sides carry an explicit expression, and check whether the user can tell from the interface which side took effect.
• Implementation side: confirm the adjudication event has a corresponding output at the user-visible layer, rather than being written only to a log.

CounterexamplesUnder-delivery — speech and gesture point at different objects, the system picks one and executes, and the interface shows no trace that the other side ever existed; over-delivery — every slight channel discrepancy pops up a modal prompt saying "input conflict detected, voice input has been adopted."

MM3-4Adjudication is revocable and can switch to the other interpretationMUST¶

In one sentence: A wrong choice can be switched back in one step, not redone from scratch.

Applies tosystems with automatic adjudication where the adjudication may not match user intent.

Rulea result produced by automatic adjudication MUST be revocable, and the system MUST provide a path to switch directly to the discarded interpretation — the user is not required to redo the entire multi-channel input. The switch path SHOULD be located in the same place as the adjudication notification, and it must remain reachable for a usable period of time after the result occurs (for the duration see mm.arbitration.revert.window). Before switching, the already-produced result must be checked against reality: the original result is undone first, and only then is the other interpretation submitted; when the original result cannot be undone, an alternative operation MUST NOT be executed on top of it. The undo applies only to this operation, preserving the user's subsequent unrelated changes. If the discarded interpretation has become unavailable because object state has changed (see MM1-4), the reason MUST be stated rather than the entry being silently removed.

Boundary conditionsthis rule does not require the candidate interpretation to be retained indefinitely; the retention duration is defined by the product. For cases already handed to the user for adjudication per MM3-5, this rule does not additionally require an undo entry.

Design applicationkeep the candidate interpretation alongside the result for a short while, so "not this one, that one" becomes a single tap rather than a re-entry. Undo and switch are two different things: the former returns to before the operation, the latter jumps to the other result.

Verification examples

• User side: after the adjudication result appears, try switching to the other interpretation, and record the number of steps needed and whether the instruction must be restated.
• Implementation side: confirm the discarded interpretation can be retrieved and executed within the retention period; construct a case where the object has changed, and confirm a reason is given.

CounterexamplesUnder-delivery — when the adjudication is wrong, the only option is to undo and then redo the entire "look at it and say move this there"; over-delivery — a permanent column of "other possible interpretations" hangs next to every result, and the interface is filled with candidate items.

MM3-5The higher the consequence, the less automatic the adjudicationMUST¶

In one sentence: High-cost conflicts are handed to the person, not picked on their behalf.

Applies tosystems where cross-channel conflict can occur on operations that are irreversible, outbound, or high-cost.

Rulewhen the operation a conflict involves is irreversible, affects other people, or falls into the high-cost tier of the product's risk classification, forbidden to automatically adjudicate and execute directly; the system MUST present the conflict to the user and let the user choose. The presentation MUST include the specific result corresponding to each side (not the channel name), so the user chooses between results rather than between channels. When the user has not made a choice, neither side MUST be executed by default. Confirmation MUST include the object, the action, key parameters, and the consequence; when risk is unknown it is treated as high-cost.

Boundary conditionssafety-critical, immediate-stop-type instructions are not bound by this rule — such instructions follow MM2-2's control path and are handled with priority within the declared safety policy. When the results of the two sides' interpretations are entirely identical, this does not constitute a conflict requiring user adjudication.

Design applicationwrite the options as "delete A" and "delete B," not as "use voice" and "use gesture" — the user does not need to know the channel, they need to know the consequence.

Verification examples

• User side: create a conflict on a high-cost operation, and confirm the system stops to ask, with the options describing results.
• Implementation side: check the correspondence between the risk-tier table and the automatic-adjudication switch, confirming the high-cost tier cannot be configured to be automatic.

CounterexamplesUnder-delivery — speech says "send to Zhang San," gaze lands on Li Si's avatar, and the system automatically picks one and sends it; over-delivery — every multi-channel operation is classified into the high-cost tier, and every interaction ends with a two-choice dialogue.

3.4 MM4 One intent executes once¶

The same user intent may be recognized once on each of two channels. Fusion grouping only solves input merging; actual submission must also withstand concurrency, retransmission, and lost result receipts. The design must prove both "one intent was formed" and "the business effect took hold only once"; merely merging the interface prompt does not prove the latter.

MM4-1The same intent produces only one side effectMUST¶

In one sentence: If two channels recognize the same thing, do it once.

Applies tosystems where multi-channel input can produce business side effects, and the same operation may be resubmitted through multiple channels, retransmission, or reconnection.

Rulewhen multiple channels within the same interaction each produce an instruction corresponding to the same user intent, the system MUST produce only one side effect. Forbidden to treat "both channels recognized it" as two expressions, and forbidden to treat it as a reason to raise confidence and execute more thoroughly (for example, escalating from a prompt to direct execution). Deduplication MUST occur before an externally visible side effect is produced; merging after the fact — once a result has already gone out, already been written, or already been notified — does not satisfy this rule.

After input merging, the submission entry MUST guarantee, before the actual side effect occurs, that the same action does not take effect again due to concurrency, retransmission, or reconnection. When something has been submitted but the receipt is lost, the outcome is "unknown"; business state is checked first. A new action identifier MUST NOT be generated to blindly retry, nor may it be displayed as failed. When a safe submission mechanism cannot be provided, the affected automatic execution path is disabled, the intent is retained, and a review / manual-handling entry is provided.

Boundary conditionsthis rule does not apply to repetition the user explicitly expresses (see MM4-4), nor to semantically complementary multi-channel input — that is two parts of one input, which only ever had one intent to begin with.

Design applicationdeduplicate at the point where intent is produced, not at the point of execution; make "fusion produces one intent" the sole downstream entry point, without leaving a bypass submission path for individual channels.

Verification examples

• User side: say a confirmation word and make a confirmation gesture at the same time, and check whether only one operation occurs.
• Implementation side: check whether any channel can submit directly, bypassing the fusion layer; construct a scenario where two channels produce output simultaneously, and confirm downstream receives only one intent.

CounterexamplesUnder-delivery — the user says "send" while also making a send gesture, and the message goes out twice; over-delivery — two real sends the user made within two seconds are both judged duplicates, and the second message is silently dropped without notice.

MM4-2Deduplication is by intent identity, not surface similarityMUST¶

In one sentence: Looking identical doesn't mean it's the same instance, and looking different doesn't mean it isn't.

Applies tosystems that perform cross-channel deduplication.

Rulethe deduplication criterion MUST be built on the identity of the intent — a comparable identity composed of the referenced object, the action, and the parameters — and it is forbidden to use identical text, identical event type, or temporal closeness as the sole criterion. Two channels' expressions of the same intent are usually not identical (one says "delete," the other makes a delete gesture); comparing text alone would miss the match. Two different intents can also be expressed identically (two consecutive "next"s); comparing text alone would misjudge them as the same. The composition of the identity MUST be explicitly defined (see mm.idempotence.key), and it MUST include enough components to distinguish two adjacent real operations.

"Semantic signature" and "interaction instance identity" MUST be kept separate: the former compares action, target domain, and parameters, and is used to judge whether two channels' expressions are saying the same thing; the latter is determined by the fusion grouping and by explicit evidence of a new operation (the user initiating again, a state change, an explicit repeat entry), and is used to judge whether this is a new instance. Identical semantic signatures alone MUST NOT be judged a repeat — two consecutive "add one"s are identical in action, object, and parameters, yet are two separate intents. An instruction lacking a bound object MUST NOT have a referring object fabricated just to satisfy the identity requirement: such an instruction uses its target domain (the current focus context, the current task) as the identity component. The actually-submitted action identifier and the input's semantic signature are defined separately, with a traceable link maintained between them; a retry of the same action reuses its identifier, while an intentional repeat generates a new action identifier.

Boundary conditionsthis rule does not require the identity to be unique across the whole system, only distinguishable within the deduplication window and scope. When a product's intent space contains no repeatable, adjacent operation of the same kind, the identity may be simplified, but the simplification must be an explicit decision. This rule does not promise that any single time threshold can separate "cross-channel duplication" from "a genuine second time": the deduplication window must reference both the arrival-delay distribution of cross-channel duplication and the interval distribution of genuine repetition; the range where the two overlap is resolved through an explicit repeat entry, state evidence, or "uncertain" handling, and it must not be claimed that one threshold necessarily separates them.

Design applicationdefine the identity as a structure, not a string: action + bound object + key parameters. Make sure intents produced by different channels land in the same identity space; otherwise deduplication has nothing to compare.

Verification examples

• User side: issue two genuinely repeated identical operations in a row, confirming both take effect; then issue one operation through two channels at once, confirming it takes effect only once.
• Implementation side: check the composition of the deduplication criterion, confirming it does not compare recognized text or event name alone.

CounterexamplesUnder-delivery — deduplicating by recognized text, the user says "next page" twice in a row and only one page turns; over-delivery — the bound object's internal content marker is also put into the identity, so the same intent fails to deduplicate because the two channels read different content markers, and it still executes twice.

MM4-3The deduplication window and scope are explicitMUST¶

In one sentence: How long it covers and what it covers must be written down.

Applies tosystems that perform cross-channel deduplication.

Rulethe deduplication time window and scope MUST be explicitly defined and verifiable (see mm.idempotence.window and mm.idempotence.scope). The scope MUST state which channels deduplication applies between, whether it crosses sessions, and whether it crosses entry points. The deduplication window and the fusion window are two separately defined quantities; forbidden to default them to being equal without an explanation — the fusion window decides "whether to keep waiting," the deduplication window decides "whether this counts as a repeat." Window values MUST have a basis; a generic value with no source MUST NOT be used. The end of the input deduplication window does not mean an already-submitted action may execute again; the retention period of submission records MUST cover the range needed for retry and receipt verification.

Boundary conditionsthis rule does not require all intent types to use the same window; setting different windows by intent-cost tier is permitted, and the tiering rule must be verifiable.

Design applicationname the two windows separately and record each one's basis separately, avoiding an engineering implementation casually sharing one constant; write the scope out clearly — participants, task, channels, and entry points — making explicit which events share one interaction.

Verification examples

• Implementation side: pull up the definitions and bases of both windows, confirming they were decided separately.
• User side: create a repeat just before and just after the window boundary, confirming the behavior matches the definition.

CounterexamplesUnder-delivery — deduplication directly reuses the fusion window's constant, so whenever the fusion window is adjusted, deduplication behavior changes with it and no one knows; over-delivery — the deduplication window is set to the entire session, so the user can execute only one operation of that kind for the whole day.

MM4-4Intentional repetition must be expressibleMUST¶

In one sentence: When the user really wants to do it a second time, deduplication must not block it.

Applies tosystems that perform cross-channel deduplication and whose intent space includes legitimately repeatable operations.

Rulethe user's intent to intentionally repeat the same operation MUST be expressible and executable. The system MUST provide at least one path not subject to deduplication suppression: this can be an explicit repeat entry, the user confirming "do it again," or a natural repetition carrying new-operation evidence. Forbidden to make deduplication a restriction the user cannot get around, and forbidden to suppress a repeat without notifying the user — for an input the user may understand as a new operation, when it is suppressed the system MUST state that it was not executed again; input from multiple channels already clearly merged into the same interaction may share one receipt without a prompt per channel.

Boundary conditionsthis rule does not require providing a quick repeat path for high-cost operations; repetition of such operations may require explicit confirmation. If the product can demonstrate that a given type of operation cannot legitimately be repeated given its semantics, recording that demonstration satisfies this rule.

Design applicationmake "a repeat was suppressed" a lightweight piece of feedback rather than no feedback at all; place the repeat entry near the result so the user does not have to go back to the starting point.

Verification examples

• User side: deliberately repeat a genuine operation within the deduplication window, confirming it either completes or the user is at least told it did not happen.
• Implementation side: check whether an execution path exists that deduplication does not affect, and whether that path requires the user to switch channels.

CounterexamplesUnder-delivery — the user wants to increase the quantity twice in a row, and the second time is silently swallowed by deduplication with no change on the interface; over-delivery — to guarantee repetition is reachable, every operation judged a repeat pops up a dialog asking "execute again?", canceling out the point of deduplication.

MM4-5When uncertain, don't silently execute a second timeSHOULD¶

In one sentence: If it's unclear whether it's a repeat, don't quietly do it again.

Applies tosystems where the deduplication criterion is inherently unable to be determined for some inputs.

Rulewhen the system cannot determine whether two occurrences of intent are one expression or two, it SHOULD default to not executing it a second time, while expressing the uncertainty: present the result of the one execution already performed along with a usable repeat entry, letting the user decide whether to do it again. Forbidden to silently execute a second time under uncertainty and offer undo only after the fact. For operations that are fully undoable and cheap to undo, the product may, per mm.idempotence.uncertain, choose the tier "execute and prominently notify," but that tier does not apply to irreversible or outbound operations.

Boundary conditionsthis rule does not require eliminating uncertainty, nor does it forbid the product from lowering the proportion of uncertain cases by adding criteria; what it specifies is the default direction under uncertainty.

Design applicationdesign "uncertain" as a legitimate value of the deduplication outcome, not an exception branch — otherwise implementations tend to pick one of two options, and usually pick "execute."

Verification examples

• Implementation side: inject three kinds of cases — missing timestamp, channel clock skew, and partially missing identity — and confirm they produce the "uncertain" tier rather than defaulting to execution.
• User side: in an uncertain scenario, confirm the user can see that only one execution occurred and knows how to execute again.

CounterexamplesUnder-delivery — the two channels' timestamps don't line up, the system cannot judge, so it executes both, and the transfer happens twice; over-delivery — as soon as uncertainty appears, the operation is aborted and the user must re-enter the whole input, so normal operation is interrupted frequently.

3.5 MM5 The task can still be completed with one fewer channel¶

Channels disappear: permission is revoked, the device is occupied, tracking is briefly lost, the environment is unsuitable, or the user turns it off deliberately. It must be made explicit how to continue after a channel is reduced, which inputs remain valid, and when recovery is allowed. The value of multiple channels includes still being able to complete the task when expressive conditions change, not merely tapping a bit less under normal conditions (R04).

MM5-1A task never has only the multi-channel pathMUST¶

In one sentence: Every task has a completion path that does not depend on the original combination.

Applies tosystems that offer multi-channel composite input.

Rulefor every task completable through multi-channel composition, an alternative completion path not dependent on that combination MUST exist. The alternative path may be slower, take more steps, or be more verbose to express, but it MUST reach the same result, and it MUST be discoverable within the product. Forbidden to have a feature triggerable only by multi-channel composition, unless that feature is physically impossible to complete with fewer channels (such exceptions MUST be explicitly declared with a stated reason). The reachability of the alternative path does not decrease because the user has turned off a channel (see MM5-4).

The alternative path must be verified through the full process of selecting the object, filling in parameters, confirming, correcting, and canceling. The input method MUST NOT be locked in based solely on the first device detected. For web drag-and-drop functionality, also check a single-pointer path that requires no dragging; providing a keyboard entry does not by itself prove a touch user can complete the task. State changes must be perceivable by assistive technology, ordinary feedback must not steal focus, and streaming candidates are not announced frame by frame. For the corresponding public basis and applicability boundary see R13, R19, R20.

Boundary conditionsthis rule does not require the alternative path to be equal in efficiency, nor does it require every remaining single channel to be able to complete the whole task on its own — what it requires is the existence of at least one path not dependent on multi-channel composition. The product MUST declare the list of available-channel subsets it commits to covering, the reachable entry point for each task under every subset, and the preservation and recovery path for when there is no operable channel at all (preserving what has already been expressed, continuing once recovery occurs). "An alternative path exists" MUST NOT be read as "any single channel is fully capable."

Design applicationwhen designing a feature, write out its single-channel path first, then consider which steps multiple channels can save; doing it the other way around tends to produce features triggerable only by composition. Existing design records recommend that optional input modes provide redundant functional coverage, so users can achieve the goal with any mode (R04).

Verification examples

• User side: go through every channel subset the product declares support for, checking whether each task can be completed; with no operable channel at all, check state preservation and the recovery entry.
• Implementation side: check the correspondence table between the feature list and its single-channel paths, confirming there is no gap, or that any gap has a declared reason.

CounterexamplesUnder-delivery — "move this over there" is the only entry point for the layout feature, so a user who cannot speak cannot adjust the layout; over-delivery — to guarantee equivalence, the multi-channel path is also broken into the same step-by-step confirmations as the single-channel one, flattening away the expressive efficiency of composite input.

MM5-2Channel unavailability is an explicit stateMUST¶

In one sentence: If it can't be used, say so — don't pretend it's working.

Applies tosystems where any channel participating in fusion may become unavailable.

Rulechannel unavailability MUST be an explicit state that the system can internally recognize and externally express, not manifest as sustained non-response or sustained low-quality results. The unavailable state MUST distinguish at least three causes: unavailable at the device or permission level, actively turned off by the user, and not applicable under current environmental conditions. Forbidden to keep counting an unavailable channel as a fusion candidate and thereby prolong the wait (see MM2-3), and forbidden to keep presenting to the user, while in the unavailable state, an appearance that the channel is available (for example, still displaying "listening").

Brief tracking loss and recovery must have a stability criterion, avoiding flip-flopping between the two interaction modes; the stabilization wait MUST NOT delay a permission revocation or a user's turning it off. Recovery only allows fresh, clearly attributed signals to participate; it does not replay old actions from the disconnection period, and it does not automatically submit a previously incomplete intent (see mm.channel.recovery.policy).

Boundary conditionsthis rule does not require the system to distinguish every technical cause of unavailability; distinguishing the three categories is the floor, because they correspond to different next steps for the user. A brief loss must also block the use of expired candidates; whether the prompt and the default channel change is decided by the declared stability condition.

Design applicationmake channel state a first-class input to the fusion engine, so the fusion logic reads the state rather than inferring it from timeout; express it externally as "currently unavailable," not "didn't hear anything."

Verification examples

• User side: revoke microphone permission and then issue a multi-channel instruction, observing whether the system states this explicitly rather than being unresponsive.
• Implementation side: inject the three cases of permission denial, user turning it off, and environmental unsuitability separately, confirming the states are distinguishable and read by the fusion logic.

CounterexamplesUnder-delivery — the camera is occupied by the system, the gesture channel never receives data, and the gesture icon on the interface stays lit anyway; over-delivery — every brief blip of a channel escalates into a "channel unavailable" banner, and prompts fire constantly during normal use.

MM5-3Degradation must be disclosed, stating what changedMUST¶

In one sentence: Quietly switching to a different behavior is worse than being unusable.

Applies tosystems where reducing a channel changes the interaction mode or changes the semantics of the same action.

Rulewhen entering a degradation path due to a reduced channel, the system MUST disclose to the user that degradation has occurred and which behaviors have changed as a result. Silent degradation is forbidden — in particular, it is forbidden to change the semantics of the same action while keeping the interface's appearance unchanged. The disclosure MUST state what the user can do now, not merely what happened. Entry points disabled or changed during degradation MUST be identifiable; they MUST NOT remain visually the same while no longer taking effect.

Boundary conditionsthis rule does not require issuing a disclosure for every fluctuation in channel state; it only requires disclosure when the interaction mode actually changes. If behavior after degradation is entirely unchanged (only a redundant channel disappeared), disclosure may be omitted, but that judgment must be verifiable.

Design applicationwrite "degradation disclosure" and "how to continue" in the same place; give controls whose semantics have changed a clear indication of the new semantics, rather than letting the user discover it through failure.

Verification examples

• User side: turn off a channel midway through multi-channel use, and observe whether a disclosure is given and whether the next step is clear.
• Implementation side: list every degradation path, confirming one by one that each has a corresponding disclosure and a statement of the behavioral difference.

CounterexamplesUnder-delivery — after the gesture channel fails, an operation that used to be "look and pinch" becomes "gaze for two seconds to trigger," the interface shows no change at all, and the user starts triggering it by accident; over-delivery — every degradation interrupts the current task with a modal dialog and requires the user to read a full statement of the capability difference.

MM5-4A separable channel that is turned off actually turns offMUST¶

In one sentence: Turning off the camera means it stops seeing — no other signal routes around it.

Applies tosystems that collect signals from two or more channels.

Rulethe user MUST be able to turn off collection channel by channel for what participates in fusion, not only as a single overall switch. Once a channel is turned off, the system MUST stop the collection, reading, or inference within that switch's declared scope of control, and it is forbidden to continue using its historical raw signal or unaccepted candidates in the current fusion. Objects and parameters already accepted are retained as task state per MM5-6; when the user separately requests deletion, it is carried out per the declared deletion scope. Forbidden to restore, without the user's explicit choice, a collection, reading, or inference use that has been turned off, by borrowing another signal (for example, automatically inferring an "equivalent gaze landing point" from head orientation after eye tracking is turned off, or reconstructing speech content from lip movement after the microphone is turned off).

The user may explicitly enable a documented alternative input: for example, actively choosing a head pointer after turning off eye tracking — this expresses pointing under the user's direct control, and it MUST NOT be presented as an inference of actual gaze, nor does it restore the data stream that was turned off. A turn-off control MUST disclose the level it actually controls (raw collection / this application's reading / a particular inference use / the entire positioning capability), and which system-necessary uses on a shared sensor fall outside this application's control. Turning it off MUST NOT cause degradation of core functionality unrelated to that channel, nor change the reachability of the alternative path required by MM5-1. Retention and deletion of data already collected before turning it off MUST both be stated and separately executable.

Turning off or revoking permission MUST cause the corresponding fragments in the pending buffer to drop out of the current fusion, and MUST prevent unsubmitted actions from continuing to use them. Fusion use, raw collection, where processing occurs, retention scope, and duration MUST be clearly stated; withdrawing one use does not mean deleting the entire history. Operational diagnostics should preferentially retain event summaries and necessary rationale; retention of raw sensitive signals requires an independent purpose and an executable deletion path, and MUST NOT be retained indefinitely on the grounds of "debugging need" (see mm.evidence.data.policy).

Boundary conditionsthis rule requires that each channel's use participating in fusion be separately controllable; a product cannot promise to turn off system-level collection it does not control. A product may combine multiple inferences on the same physical sensor into a single switch, but that combination must have a technical basis and state exactly which collection and which uses that switch actually turns off; physically sharing a sensor does not automatically cancel separate control over this application's individual uses.

Design applicationmake "stop collection" and "delete existing data" two separately executable operations, each stating its consequence; the off state must take effect at every entry point, including passive channels and background collection.

Verification examples

• User side: after turning off one channel, observe whether system behavior stops changing with that channel's signal.
• Implementation side: check for any path that reconstructs an equivalent capability from another channel's signal; confirm the off state is actually read by the fusion layer, not merely recorded in settings.

CounterexamplesUnder-delivery — the user turns off eye-tracking collection, and the system switches to estimating the gaze landing point from head orientation, with fusion behavior almost unchanged; over-delivery — turning off any single channel disables all multi-channel capability entirely, without providing the alternative path MM5-1 requires, so the switch becomes a deterrent.

MM5-5Environmental and social conditions factor into availabilitySHOULD¶

In one sentence: Being able to speak doesn't mean it's convenient to speak right now.

Applies tomulti-channel systems used in mobile, public, or shared environments.

Rulethe judgment of channel availability SHOULD include environmental and social conditions, not only device- and permission-level technical availability: noise, occupied hands, a public setting where it is inconvenient to speak, or people nearby making large gestures inconvenient, can all make a technically available channel practically unavailable. The system SHOULD adjust the default channel combination accordingly when such conditions can be observed, and it is forbidden to interpret a user not using a channel as that channel not being needed, and permanently lower its weight on that basis. The basis for judgment MUST be resolvable and overridable by the user — a channel combination the user explicitly chooses takes priority over the system's contextual judgment. A bystander's speech and actions MUST NOT be treated by default as the current user's instructions.

Boundary conditionsthis rule does not require the system to have environmental-sensing capability; when it does not, handling per the user's explicit settings is sufficient, with the limitation recorded. This rule does not require inferring the details of the user's social situation, only that directly observable conditions be factored into default selection.

Design applicationmake the result of contextual judgment a "default combination suggestion" rather than a "forced combination"; let one explicit user choice override it and persist across similar contexts.

Verification examples

• User side: initiate the same task under a noisy environment and a quiet environment separately, observing whether the suggested default channel combination is reasonable and can be overridden.
• Implementation side: confirm the user's explicit setting takes effect above contextual judgment, and is not modified back by usage frequency.

CounterexamplesUnder-delivery — in a meeting room the system still treats voice as the only convenient path, and the user can only whisper it three times over; over-delivery — the system frequently auto-switches the primary channel based on ambient noise, so every time the user raises a hand they no longer know whether to speak or point this time.

MM5-6Switching channels does not reset what has already been completedMUST¶

In one sentence: Continuing in a different way is not starting over from the beginning.

Applies tosystems that allow the user to switch input channels within one task.

Rulewhen the user switches channels partway through a task, the part already expressed and accepted by the system MUST be preserved; it is forbidden for a channel switch to reset input state, clear an already-bound object, or require the user to restate prior content. After switching, the system MUST be able to state what is currently still missing. Channel switching is a common means of error recovery — a user tends to switch to a different channel and try again after a recognition failure on one channel (R04) — so the switch path MUST remain reachable after a recognition failure, and the context MUST NOT be cleared because of the failure.

Boundary conditionsa slot that has genuinely become invalid is marked individually with a stated reason, while the other valid parts are retained. Accepted content that cannot be directly displayed on the new channel MUST be provided in a retrievable form; it MUST NOT be deleted just because it cannot be displayed. This rule does not require indefinite or cross-session retention.

Design applicationattach task state to the task, not to the channel session; make "what's still missing" a current state queryable at any time, not a prompt that appears only on failure.

Verification examples

• User side: speak halfway through, then switch to finishing with a gesture, and check whether the first half is still valid.
• Implementation side: at the channel-switch point, check whether the task-state object is rebuilt; construct a scenario of switching after a recognition failure, confirming the context is preserved.

CounterexamplesUnder-delivery — after speech says "move this to" and recognition fails, switching to touch reveals the bound object is also gone, so the user must start over from scratch; over-delivery — the completed part is preserved but the user is not told what was preserved, so the user expresses it again, and the system treats it as two intents (also violating MM4-1).

3.6 MM6 Output is divided by channel¶

The fusion problem on the output side is symmetric to the input side: a single system output can be distributed across visual, auditory, tactile, and other channels. This principle governs that distribution — who carries what, when they appear together, and what happens with one fewer channel. A common mistake is understanding multi-channel output as multiple layers of insurance: the same sentence gets written on screen once, spoken aloud once, and then a buzz on top — the result is three channels competing for attention, and the user catches none of them. Existing records of human multi-channel expression note that the natural organization is complementary rather than redundant — different channels carry different semantic components, and the proportion of exact repetition is very low (R04).

MM6-1Information that cannot be missed lands on a channel that can receive itMUST¶

In one sentence: When the screen isn't in view, don't put what matters only on the screen.

Applies tosystems where output can be distributed across multiple channels, and where information exists that the user must not miss.

Rulefor information the user must not miss — receipts that an operation has been executed, irreversible results, errors, and states requiring user intervention — the system MUST deliver it on a channel judged receivable under currently observable conditions, and MUST explicitly designate the carrying channel for such information rather than letting it scatter with the default layout. The choice of carrying channel MUST be based on observable receipt conditions (whether the screen is in view, whether audio is playing aloud, whether a device is worn) and the user's output-channel settings. "Sent," "accepted by the platform," "presented," and "processed by the user" are four different things, and none MUST be passed off as another; channel routing is the product's decision and does not constitute a guarantee that the user has been informed.

When the receipt condition is unknown, or none of the candidate channels is known to be available, the system MUST retain this information as "receipt unconfirmed," provide a reviewable or pending path, and escalate in a bounded way per predefined rules (switching to another channel, increasing prominence, retrying later); forbidden to place such information only on a channel that is currently unreceivable, forbidden to treat "sent" alone as proof of delivery, and forbidden to restore a channel the user has turned off, purely to satisfy this rule. Explicit confirmation is required only when subsequent processing must depend on the user's decision; the user MUST NOT be asked whether they saw it after every single reminder. Safety-critical information MUST also provide reviewable details.

Boundary conditionsthis rule does not require every piece of information to be delivered on multiple channels; it applies only to the "must not miss" tier. The definition of that tier is decided and recorded by the product, and the classification itself must be verifiable.

Design applicationfirst label every output type "can miss / cannot miss," then decide the channel assignment; write the carrying channel into the output definition (see mm.output.critical.carrier) rather than leaving it to the rendering layer's default rules.

Verification examples

• User side: trigger the receipt of an irreversible operation while the user's gaze is off the screen, and check whether the user learns of it.
• Implementation side: check the correspondence between the must-not-miss information list and its declared carrying channel, confirming nothing is omitted.

CounterexamplesUnder-delivery — the confirmation receipt in a headset appears only as a small line of text on a panel the user isn't currently looking at, so the user thinks it didn't execute and does it again; over-delivery — every system message is upgraded to must-not-miss and pushed simultaneously with sound and haptics, so important information drowns in the reminders.

MM6-2Division of labor is the default; redundancy needs a reasonSHOULD¶

In one sentence: Three channels saying the same sentence is not three times the assurance.

Applies tosystems where a single output can be distributed across multiple channels.

Rulea single output presented across multiple channels SHOULD divide different components across them — for example, vision carrying detail and reviewable content, hearing carrying the conclusion, touch carrying the mere fact that an event occurred — rather than repeating the same content on every channel. Adopting redundancy (repeating the same content across multiple channels) SHOULD have an explicit, recorded reason; the typical valid reason is that the receipt condition is uncertain or the information falls into the must-not-miss tier (see MM6-1). Forbidden to use redundancy as the default fallback when no division-of-labor decision has been made. The division-of-labor plan MUST state how the information remains complete when any given channel is missing (see MM6-4).

Boundary conditionsthis rule does not forbid redundancy, nor does it require the division to be broken down to every semantic component; what it requires is that both division of labor and redundancy be decisions that were actually made. The equivalent alternatives accessibility requires are not the redundancy this rule refers to — that is a requirement of MM6-4.

Design applicationwrite a small table for each output type: which part of this information goes to the eyes, which to the ears, which to the skin, and why. A neutral haptic cue can signal that an event occurred; the specific object, quantity, and result should have a clear, obtainable explanation.

Verification examples

• User side: observe one typical output, record what each of the three channels says, and check whether they are three copies of the same sentence.
• Implementation side: spot-check output definitions, confirming redundant items carry a stated reason.

CounterexamplesUnder-delivery — every notification pops up, reads the full text aloud, and vibrates all at once, so the user turns off two of them and then misses the one that actually mattered; over-delivery — to avoid repetition, the conclusion is placed only on the auditory channel, so a muted user gets no conclusion at all (also violating MM6-4).

Basis and referencesnatural human multi-channel expression is predominantly complementary, with different channels carrying different semantic components and a very low proportion of exact repetition (R04). That record comes from an input-side observation; applying it to output-side division of labor is a design inference of these guidelines and does not constitute proof that "division of labor is generally better than redundancy" — on the output side, under what conditions redundancy is preferable still has no conclusion these guidelines can cite; the product must verify this itself.

MM6-3Cross-channel output timing is definedMUST¶

In one sentence: Too far apart, and people treat it as two separate things.

Applies tosystems where a single output is presented across two or more channels.

Rulefor a multi-channel presentation belonging to the same output, the relative timing MUST be defined: which one appears first, what interval range is allowed, and how to handle exceeding that range (see mm.output.sync). The interval value MUST be measured and recorded by the product against the target context, and it is forbidden to use a generic value with no source. When the defined range is exceeded, the system MUST avoid letting the user understand them as two independent events — it may merge them, or drop the late-arriving one and keep the already-presented one intact, but it MUST NOT present them one after another unmodified without any handling.

Every channel's output MUST be linked to the same business event and its current result. When the result is corrected or canceled, an expired output not yet played MUST be removed, and an already-presented wrong conclusion MUST be corrected; a "send failed" screen MUST NOT be allowed to appear followed by a queued "send succeeded" voice line. Canceling playback only stops presentation; it does not automatically undo an already-executed business action (see mm.output.update.policy).

Boundary conditionsthis rule does not require every channel to be strictly simultaneous; different channels naturally have different presentation durations, and this rule constrains the starting sequence and the acceptable interval. A deliberately designed order (a cue first, elaboration later) is not a synchronization failure, but its order and interval must equally be defined.

Design applicationmodel cross-channel output as one composition with timing constraints, not several independent dispatches; write the handling for a late-arriving channel in advance, rather than leaving it to a runtime accident.

Verification examples

• User side: under an artificially added delay on one channel, observe whether the user understands the output as two separate things.
• Implementation side: inject a single-channel rendering delay, confirming the defined handling is executed rather than presenting it unmodified.

CounterexamplesUnder-delivery — the haptic feedback happens immediately, and the corresponding visual explanation only appears seconds later, so the user thinks these are two different events; over-delivery — to guarantee synchronization, every channel is aligned to the slowest one, and immediate feedback becomes sluggish across the board.

Basis and referencesthis rule's requirement that "the synchronization window is defined and measured by the product itself" is a design position of these guidelines, holding regardless of external research — without measurement there is no verifiable commitment. The qualitative statement often cited, that "a person's judgment of whether cross-channel signals belong to the same event has a time window, and the window's width varies by person and condition," is registered as R09, for which an access record exists but the body text has not been obtained, and it remains pending verification; until the original text is verified, this rule does not cite it in the tone of "established by research," nor does it set any millisecond value on that basis.

MM6-4Any single channel is understandable on its ownMUST¶

In one sentence: With one fewer channel, information may shrink but must not become wrong.

Applies tosystems that present output divided across channels.

Ruleoutput divided across channels MUST achieve this: when the user can only receive any one of the channels, the information obtained is still correct — it may be less or coarser, but it MUST NOT produce an understanding opposite to or ambiguous relative to the complete information. Key components that change the direction of meaning (negation, quantity, object attribution) MUST appear in whichever channel expresses that conclusion; it is forbidden to place the conclusion on one channel while placing the component that reverses its direction on another. A neutral event notification may omit these components — a haptic cue that expresses no conclusion is not disqualified for not encoding negation and quantity; but in that case the necessary detail MUST be obtainable through a currently available path; it must not be left as a mere notification with no way to trace the detail. When information falls into the must-not-miss tier, it must also simultaneously meet MM6-1's carrying requirement. This rule's relationship to MM6-2: MM6-2 says division of labor is the default; this rule sets its floor — division of labor MUST NOT be split such that any single channel, viewed alone, is wrong.

Boundary conditionsthis rule does not require every channel to convey complete information; what it forbids is a wrong understanding from a single-channel perspective, not missing information. Three checks must be carried out separately: whether each channel is consistent with the true state (this rule), whether necessary information is reachable through a currently available path (MM6-1), and whether an accessibility-equivalent alternative is provided (per separate accessibility requirements). "Vision displays the reason it was not sent + one neutral haptic reminder" satisfies this rule; "only a neutral reminder with no way to obtain the error detail" violates MM6-1, not this rule. Purely atmospheric or decorative output carries no semantics and is outside the scope of this rule.

Design applicationcheck whether each channel's output can stand as an independent sentence: hearing only the sound, seeing only the visuals, feeling only the vibration — none of them should produce an opposite conclusion. Pay particular attention to which channel negation words and quantity words land on.

Verification examples

• User side: block each output channel in turn, ask the user to restate the information received, and check whether an understanding opposite to reality occurs.
• Implementation side: spot-check output containing negation or quantity, confirming the key component was not split off to a single channel.

CounterexamplesUnder-delivery — the screen shows "not sent," while the voice reads only "sent," so a muted user and a user not looking at the screen reach opposite conclusions; over-delivery — to guarantee every channel is independently understandable, every channel fully repeats all the content, degenerating into the default redundancy MM6-2 forbids.

MM6-5Output channel division of labor resolves by context and is user-adjustableSHOULD¶

In one sentence: Who carries what is decided by the current context, and people can change it.

Applies tosystems where the availability or suitability of output channels varies with context.

Ruleoutput channel division of labor SHOULD resolve by current context — device form, whether audio plays aloud, whether it is worn, whether the screen is shared, whether do-not-disturb is active — rather than being fixed to a single channel. Users SHOULD be able to change their division-of-labor preference (for example, "show me the conclusion too"), and that preference SHOULD persist across similar contexts. Forbidden to interpret a user turning off one output channel as license to also discard the information it used to carry — information carried by a turned-off channel MUST be carried instead by another channel, or the user MUST be explicitly told it is no longer provided. Routing MUST respect system mute, do-not-disturb, and user off-settings; sensitive detail MUST NOT automatically move to a speaker or a shared screen just because the original channel is unavailable.

Boundary conditionsthis rule does not require offering per-item channel selection; a category-level preference is sufficient. A product without context-sensing capability handles this per user settings, with the limitation recorded.

Design applicationwrite the division of labor as a resolvable rule of "information category × context → channel," letting the user preference enter the same rule as an overriding layer, rather than as separate switches scattered across individual features.

Verification examples

• User side: trigger the same kind of output under both a speaker context and a headphone-worn context, observing whether the division changes and is reasonable.
• Implementation side: turn off one output channel, confirming the information it used to carry is now carried by another channel or explicitly disclosed.

CounterexamplesUnder-delivery — the user turns off voice playback, and the conclusion that used to be carried only by voice never appears again, not even on the interface; over-delivery — the division-of-labor rule changes frequently with context, so the same class of message appears in a different place every time, and the user can never form an expectation.

MM6-6Multi-channel output does not stack into overloadSHOULD¶

In one sentence: Three things arriving at once is the same as receiving none of them.

Applies tosystems that may produce multiple outputs within a short time.

Rulethe system SHOULD set a ceiling on the total volume of multi-channel output within the same time window, and above that ceiling merge, delay, or drop by semantic importance rather than presenting everything. A ceiling MUST exist in some form: an equivalent total-volume control measure may be substituted for a recorded reason (for example, rate-limiting separately by semantic category), but no total-volume control at all is not permitted; the measure adopted and the argument for its equivalence MUST be recorded and subjected to the same verification as setting a ceiling. Forbidden to assume channels don't interfere with each other just because they're different — vision, hearing, and touch arriving together still jointly occupy attention. The merging rule SHOULD preserve must-not-miss information (see MM6-1), preferentially dropping the parts that can be missed.

Boundary conditionsthis rule does not specify a concrete total-volume figure; the ceiling is defined by the product per context, with the basis recorded. Safety-critical alerts are handled by a dedicated alerting policy and are not dropped because of the ordinary budget; that policy must also define priority and alert-storm handling, and unlimited stacking is not permitted.

Design applicationmake the output total a budget shared across channels, rather than each channel counting on its own; rank by importance first and then allocate the budget, not the other way around.

Verification examples

• User side: construct a dense-event scenario, and check whether the user can identify the most important item among them.
• Implementation side: confirm a cross-channel shared output budget exists, and that the must-not-miss tier is preserved during merging.

CounterexamplesUnder-delivery — five notifications arrive at once, and the interface pops five cards, reads five passages aloud, and vibrates five times; over-delivery — the ceiling is set too strict, merging away even the immediate feedback for the user's own active operations, so actions stop getting a response.

4. Terminology and definitions¶

This chapter distinguishes configuration, the facts of a single interaction, and user-visible feedback; state and actual input are not Tokens.

4.1 The factual contract of one interaction¶

The table below is for implementing and reviewing MM1–MM6; it does not require adopting a specific protocol, nor does it add new Tokens. The actual events, objects, coordinates, times, and receipts are operational data.

Collection state (collecting / closed), interpretation state (complete / missing item / conflict / unknown), and execution state are stored separately. For example, "collection closed, target location missing, not submitted" is a legitimate state; the interface should display "card A selected, please choose a target column," not "task failed."

Operational records are retained at a minimum per mm.evidence.data.policy; reviewability does not require retaining every raw signal.

Appendix A: Fault-injection verification checklist and classification check¶

This checklist is used to verify whether a clause actually takes effect; it adds no new obligations. Inject item by item and record the system's actual behavior; "not applicable" must be accompanied by a scenario-based rationale; "not tested" is recorded as pending verification and must not be recorded as passed.

A.1 Reference and binding

A.2 Time and the fusion window

A.3 Conflict and adjudication

A.4 Repetition and idempotence

A.5 Channel availability and degradation

A.6 Output division of labor

A.7 Composite chains and normal-use cost

Each use case records the design definition, mechanism evidence, and user understanding separately; passing a documentation check does not equal passing actual product testing.

A.8 Classification check

Used to test whether Chapter 1's division holds: take 10 to 15 concrete requirements (which can come from clauses in these guidelines or from real review comments), and have at least three reviewers who did not participate in the writing independently judge which principle each belongs to. If assignment disagreement concentrates between two particular principles, that shows the objects of regulation for those two principles have not been separated — in that case the principles should be adjusted, rather than adding an intermediate layer or a mapping explanation. The two places known to need priority testing are made explicit in Chapter 1 (MM2 and MM4, MM3 and MM5). The number of reviewers and the disagreement criterion here are an internal check method these guidelines recommend, not a standard method validated by literature.

Appendix B: Evidence boundaries and source types¶

B.1 The criterion for normative terms

The sole basis for marking something "MUST" is: without it, some commitment to the user would fail under a foreseeable condition. The three types of argument below provide different kinds of support; they are not three independent sources of obligation, and an implementation reference by itself is not sufficient to decide "MUST" —

The eight rules marked "SHOULD" are MM1-6, MM2-5, MM2-6, MM4-5, MM5-5, MM6-2, MM6-5, and MM6-6, and each of them contains a forbidden-level clause (for judgment see 2.2). The "should" clauses within them permit deviation with a basis; the "must / forbidden" clauses remain the floor and cannot be exempted by a deviation record.

B.2 The three places where this specification's evidence is thinnest

Listed explicitly, not concealed under the tone of a clause:

1. The MM2 group's window values have no universal baseline that can be directly adopted. The temporal-relationship records in the listed sources come from specific channel combinations (speech and pen, speech and gesture) and a specific task (map-type spatial tasks); their magnitude cannot be directly transplanted to combinations such as gaze and pinch, or speech and touch. These guidelines therefore only require that "the window be explicitly defined, have a basis, and be verifiable," without giving a value — this is also the group of clauses in these guidelines most likely to be satisfied with mere formality.
2. The MM6 group's division-of-labor principle comes from an input-side observation. The "complementary rather than redundant" record describes how people organize multi-channel information when expressing themselves (R04); applying it to system output-side division of labor is a design inference of these guidelines, not a direct transplant of the source's conclusion. The effect of output-side division of labor still awaits verification by each product.
3. MM3's adjudication rule lacks a cross-product comparison of effects. The listed platform documentation records a written-down input-priority statement (R16, R17), but no public result was found comparing different adjudication strategies on user comprehension and correction cost. These guidelines therefore only require the adjudication rule to be declarable, reproducible, and revocable, and do not recommend any specific priority order.

B.3 What these guidelines do not do

They give no fusion architecture (signal-level, feature-level, and semantic-level are all left unspecified), no recognition threshold or confidence setting, no time-window value, no channel priority order, no reference-resolution algorithm. These are decisions for the product and the domain; these guidelines only specify that such decisions must be made, must be verifiable, and which values are not permitted. Recognition accuracy, device perceptibility, and task safety still require independent measurement; they cannot be inferred from the composite behavior being compliant.

B.4 Sources

For the complete source cross-reference and verification scope see reference.md. Clauses in these guidelines do not hold simply because some product has done it this way; a product's practice is evidence that "this kind of mechanism is feasible in a real product," not a basis for "this is how it should be required."

Appendix C: Determining values for the fusion window and the deduplication window¶

This appendix is an application reference for MM2-1, MM2-3, MM4-3, and MM6-3; it adds no new obligation and gives no recommended value.

First state clearly which side's users are being protected, then choose the value. Measurement records should include the sample's population composition, task type, and device conditions; when the population, task, or device changes, the value must be reconfirmed.

---

Implementation acceptance scenarios¶

The scenarios below turn existing clauses into reviewable acceptance inputs; they set no separate universal performance threshold. Select by the product's applicable capability, and supplement with real devices, users, input sequences, and evidence; record the reason when not applicable, and an item not executed must not be recorded as passed.

Each scenario checks the configuration's effective value, the execution record, and the user-understandable result separately. Retain the version, the target, the event time, the failure scope, and the recovery result; when the external result is unknown, it is not recorded as either success or failure.