Sampling and dropped reports systematically undercount rare actions

What it is

To save bandwidth and storage, clients often sample 1% or 10% of users or events before upload; network failure, process death, and queue overflow then drop reports at random or in ways tied to the environment. Frequent actions still leave a recognizable shape. For crashes, payment failures, or an accessibility shortcut—rare actions—a single missed draw or a single failed send can move the count from one to zero. Incidence of the rare is then systematically pushed down, and it looks as if “almost no one does this.”

Why it happens

Bernoulli sampling is unbiased for expected counts, but for zero-inflated rares most windows observe zero, variance is huge, and small-traffic estimates hug zero. If the sampling unit is the user, rare actions on never-sampled people are gone forever; if it is the event, a burst of failures can vanish as a block. Upload loss correlates with weak networks, low-end devices, and background kills, environments where errors and interrupts are already more common—loss is positively related to the target behavior, so undercount is not noise. Analyzing “users who have data” then conditions on a sample whose tail has already been shaved.

Studying it

Document sampling rate, sampling unit, retry and on-device buffer policy, and whether loss is observable (failed-send counts, queue length). Stop estimating rare incidence from a sampled stream; switch to a full census, a higher sampling rate, or an authoritative server log. Estimate loss with capture–recapture or a server-side match, stratified by network and device, to see whether loss tracks errors. When reporting a rare rate, give effective exposure and the number of zero-event windows, so “not seen” is not written as “rate equals zero.”

Where it stops holding

For behaviors that already happen to nearly everyone, light sampling’s shape error can be ignored. Server-side posted transactions usually do not depend on client sampling; client loss hurts process events, not the ledger. Intentionally down-sampling debug logs is an engineering choice if rare alerts travel a full-census channel. Privacy minimization limits how long a census can be kept, but sampling is not a substitute for having observed the tail.

Applying it

• Mark crashes, payment failures, permission denials, and accessibility shortcuts as census events; keep them out of percentage sampling buckets.
• Label dashboard series as “sampled estimate” or “full count”; never use a sampled series to claim an error has vanished.
• Watch upload failure rate and per-device arrival; freeze rare-event conclusions when failure rises.
• Before a postmortem on “why no one uses this,” check whether the action sat in a sampling bucket; if it did, fix capture before drawing a product conclusion.

Related

• Same group: Q3.11.1 Logs record behavior but not motive · Q3.11.2 Instrumentation design determines which questions can later be answered · Q3.11.3 Missing events cannot be backfilled after the fact · Q3.11.4 Inconsistent event names prevent joining data across versions · Q3.11.6 Ad blockers and privacy settings permanently omit some users · Q3.11.7 The same event may fire under unequal conditions across platforms
• Adjacent: Q1.04 Sampling and representativeness · Q1.08 Sample size
• Search terms: sampling undercount · rare-event logging · client-side sampling