Having several people repeat the same task and comparing answers is the base quality control

What it is

Crowdsourcing comes with no employment-era skill guarantee; the most basic substitute is redundancy: the same task goes independently to several people, and answers are compared — agreement passes, split goes to review. The reasoning is an informal law of large numbers: any one stranger's error rate is uncontrollable, but multiple independent errors cancel each other with rising headcount, and the credibility of the agreed result rises with it. The simple form is majority voting; advanced forms weight by each worker's historical accuracy or use statistical models to estimate every worker's reliability and each item's truth jointly. Quality control here is not screening people but buying truth with redundancy.

Why it happens

Redundancy works on the premise of independent errors: the probability that two people err identically is the product of their individual error rates, decaying exponentially with the number of copies, so "the majority is wrong" collapses and the majority vote converges to truth. The premise also exposes the mechanism's soft flank — correlated errors break the decay (a dedicated entry develops that theme; not here). Progress in aggregation technology is all about efficiency under the independence assumption: naive majority voting weights all votes equally, wasting the information in "some people are historically more accurate"; weighted and model-based aggregation (such as expectation-maximization iterating between worker reliability and item truth) converts the same redundancy into higher accuracy — the algorithm substituting for part of the human redundancy. Redundancy also dilutes bad actors in passing: random sloppy answering drowns against a majority of genuine ones unless attackers coordinate at scale — anti-laziness becomes a probability question, not a virtue question.

Studying it

• Paradigm: annotation experiments with ground-truth benchmarks — fix a set of inputs with known answers, vary redundancy and aggregation method, and measure the accuracy curve of aggregated results; re-analysis of platform logs (replaying aggregation strategies on historical redundancy from published tasks) is a low-cost second route.
• Variables: number of redundant workers, aggregation method (majority / weighted / model-based), and worker quality mix as independent variables; aggregated accuracy, minimum redundancy reaching a target accuracy, and effective unit cost as dependent variables.
• Use in interface research: aggregation configuration for crowdsourcing platforms — redundancy presets, automatic routing of split items (re-dispatch or escalate) so human effort lands only on disagreement.
• Methodological caveat: the gold standard itself has measurement error, and "agreement with gold" underestimates aggregation quality when the gold is shaky; accuracy curves saturate quickly on easy tasks and climb slowly on hard ones, so extrapolating a minimum redundancy across task types is a standing error.

Where it stops holding

Redundancy handles independent random errors well and systematic bias not at all: with ambiguous instructions everyone errs identically, and any level of agreement is worthless (that layer belongs to its own entry, one nod here) — correction must return to the instruction layer. Cheap per-item classification suits redundancy; creative tasks (copywriting) have no "same answer" to compare, redundancy fails, and review replaces it. Redundancy also diminishing-returns: the credibility gain from one to three workers far exceeds seven to nine, while cost is linear — the optimum is the cutoff of the cost–accuracy curve, not "more is safer."

Applying it

• Set redundancy tiers by the task's single-worker accuracy: the more accurate one worker is, the fewer copies needed; for new task types, run a small sample to estimate single-worker accuracy before tiering.
• Never ship a single aggregation strategy: agreement auto-passes, splits auto-requeue or escalate — spend human attention only on disagreement.
• Add weighted aggregation once history exists: weight workers by past agreement with majority/gold, converting the same redundancy into higher accuracy.
• Tilt redundancy toward high-value items: more copies where errors are costly, single copy plus spot audits where they are cheap.
• Verification: replay accuracy–cost curves across redundancy levels and aggregation strategies on a gold subset, pick the cutoff, then track split rates live — split rates rise before quality incidents and are the leading indicator that tiers need raising.

Related

• Same group: V9.05.2 Redundancy scales cost linearly and must be tiered by task difficulty · V9.05.3 Seeded gold questions continuously estimate contributor accuracy · V9.05.4 Agreement does not guarantee correctness; shared misconceptions err in unison · V9.05.5 Paying per completed item induces fast, low-quality work
• Nearby: V9.04 Crowdsourcing Task Decomposition and Instructions · V9.01 Voting and Consensus Mechanisms
• Search terms: majority voting · redundancy · Dawid-Skene