Q3.22
Multiple comparisons and result picking
Cards in this group · 4
- Q3.22.1Testing many metrics at once raises the chance of a false positive
- Q3.22.2Multiple comparisons need a corrected threshold or a primary metric named in advance
- Q3.22.3Reporting the significant slices from many breakdowns is data peeking
- Q3.22.4The primary metric is locked before the experiment starts, not chosen afterward