Reporting the significant slices from many breakdowns is data peeking

What it is

Cutting one metric by country, device, new versus returning, channel, weekday, and then writing only the cells that came out significant into the conclusion, is data peeking in the slice-picking form. The cells are looked at during analysis; the significant ones are kept and the rest discarded as “irrelevant.” The aggregate may hold nothing; the slices will still produce a pretty cell. This is not the same table as testing many different metrics at once: what is repeated here is many cuts on one endpoint. It is also not the move of taking a pattern found in exploration and dressing it as already confirmed on the same data—slice-picking fails at selection in the report, at “which cell gets told.”

Why it happens

Each cut is an extra test, and the cutting rule can still change after seeing where the green is (different age bins, two countries merged). Selection eats the most extreme noise, point estimates inflate, nominal p fails. The audience sees the chosen cell and not how many cells were searched, so a highlight found by search is heard as a pre-cared-about segment effect. Interaction or heterogeneity can be a real question; a real question is asked with pre-declared cuts, a bounded number of cells, and a test aimed at the interaction, not by walking a heatmap in the meeting. Peeking does not require opening the data many times: computing every slice once and then picking is already enough.

Studying it

List allowed breakdowns, the reason for each cut, and the interaction test that will be used; the cell count enters multiplicity. Every other cut is exploratory, and the report shows the search space (what was cut, how many cells, how many cleared a line). A claimed slice effect is confirmed on a new sample or a held-out fold; without new data it stays exploratory. An interaction test beats “run the main effect inside every level and keep the significant ones.” Robustness: change the bins, drop the largest cell, see whether the story remains; a slice that vanishes on a recut does not get promoted.

Where it stops holding

When the product only serves one pre-defined population (one language, one device), that cut is not peeking, provided it was written in the plan and not in the results. Extremely sparse cells that look significant are often variance exploding, not a find. Showing every slice descriptively, with no picking narrative, can be monitoring if the language stays “not yet confirmatory.” Safety events cut on a prewritten list so harm is not missed; still report how many items were cut, rather than pasting only the red one.

Applying it

• The conclusions page may not show an audience label that was absent from the plan. To talk about a slice, promote it from exploration to the next confirmatory round.
• Exploratory appendices include the full slice matrix, not a few attractive bars.
• When a meeting finds that “this device is very significant,” log it as a hypothesis to test; do not rewrite the current release note.
• Check: count the cells actually cut. A page that reports only the significant ones, with no denominator, is void.

Related

• Same group: Q3.22.1 Testing many metrics raises false positives · Q3.22.2 Correct the threshold or name a primary in advance · Q3.22.4 The primary is locked before launch, not chosen after
• Adjacent: Q3.21 Statistical significance and effect size · Q1.02 Exploratory and confirmatory research
• Search terms: data peeking · subgroup fishing · slice picking