Instrumentation design determines which questions can later be answered

What it is

The questions data can later answer do not extend beyond the events, properties, and identifiers declared at the time. Instrumentation design draws that question space: whether “filter applied” exists, whether the chosen filters are properties, whether a session can be stitched. A question that was not designed in is not “not yet analyzed”; the instrument cannot see it. Stockpiling clicks and asking research questions afterward lets the warehouse shelves decide what can be sold.

Why it happens

Each event is a cut on the world: trigger, payload, primary key, timestamp. State changes outside the cut might as well not have happened. Cardinality and grain of properties decide how far a slice can go—category questions need a category field; treatment effects need an experiment key. Which of session, user, and device identifiers stay stable decides which unit of analysis can stand. Teams typically instrument the few numbers the current dashboard wants, so a question six months later hits an empty field. Question space is a design product, not a property that appears after data accumulate.

Studying it

Work backward from research questions and decisions: which answers must be distinguished, which state changes must be visible, which keys must travel with them. Write candidate questions as “with this event and these properties, we can estimate X; without them, we stop at Y.” Review the event table for empty properties, unjoinable identifiers, and events that have a numerator but no denominator. Walk a short real task: does every critical action have an event, and can the path be reconstructed later. Accept the instrumentation against those questions, not against a rising event count.

Where it stops holding

Not every imaginable question deserves client performance and privacy budget; question space should be deliberately smaller than the possible world. Early exploration can log coarse events if it is admitted that fine questions are unanswerable. Default events in a third-party analytics suite are that suite’s question space, not the product’s. Fields the law forbids collecting permanently block a class of questions and should be deleted from the research plan, not postponed as something to “figure out after launch.”

Applying it

• Open an instrumentation review with “three questions next quarter must be able to answer,” and check event by event whether the schema can carry them.
• Write an expected path for each critical task and map a step to an event; missing steps get events, not a later page-view proxy.
• Refuse “log it for now” packages that lack a property dictionary and an identity strategy.
• After launch, run each of those three questions; if they cannot be queried, instrumentation is unfinished, not analysis.

Related

• Same group: Q3.11.1 Logs record behavior but not motive · Q3.11.3 Missing events cannot be backfilled after the fact · Q3.11.4 Inconsistent event names prevent joining data across versions · Q3.11.5 Sampling and dropped reports systematically undercount rare actions · Q3.11.6 Ad blockers and privacy settings permanently omit some users · Q3.11.7 The same event may fire under unequal conditions across platforms
• Adjacent: Q6.02 Goals–signals–metrics · Q1.01 Framing research questions
• Search terms: instrumentation-limited question space · event taxonomy · analytics schema