Redundancy scales cost linearly and must be tiered by task difficulty

What it is

Redundancy is the workhorse of crowd quality control, and every copy is billed per item: three-way redundancy is triple unit cost, five-way is quintuple. The quality budget is therefore not a decree of "uniform N copies" but a decision of tiered configuration by task difficulty — tasks with high single-worker accuracy need two copies, low-accuracy tasks may need five, and spending hard-task redundancy on easy tasks burns the whole budget. The redundancy tier — the lookup table translating difficulty into copies — is the highest-leverage cost knob in crowdsourcing operations.

Why it happens

The benefit side of redundancy follows a saturating curve while the cost side is a straight line. The lower the single-worker accuracy, the more copies needed to aggregate to a target confidence, with diminishing increments — one-to-three buys far more credibility than five-to-seven — while each added copy costs strictly linearly. The intersection of the two lines sets the optimum, and the intersection moves with single-worker accuracy: harder tasks (less accurate workers) shift it right and steepen unit cost; easier tasks shift it left to "two copies, compare, requeue on split." Difficulty itself is not uniform — a batch typically mixes difficulty bands, with edge inputs (genuinely ambiguous judgments) a minority contributing most of the disagreement. So tiering is not a binary hard/easy but routing the stream by predicted difficulty: clear band low redundancy, contested band high, splits escalate automatically. Mis-tiered costs are asymmetric: under-redundancy leaks bad results (quality incidents), over-redundancy burns budget (cost incidents) — and the latter's invisibility makes it the more commonly ignored failure.

Studying it

• Paradigm: tier simulation by gold replay — on ground-truthed historical data, simulate tiering strategies (uniform N / difficulty-tiered / dynamic re-queue) on total cost and final accuracy to locate the Pareto front; online experiments randomize tiers to verify external validity of the simulation.
• Variables: tiering strategy, difficulty-prediction features (historical split rate, share of edge judgments), and target accuracy threshold as independent variables; total cost, final aggregated accuracy, and effective cost per item as dependent variables.
• Use in interface research: task publishers turning redundancy from a global constant into a per-batch parameter, with suggested copy counts from difficulty prediction.
• Methodological caveat: difficulty features suffer cold start (new tasks have no history); predicting future difficulty from historical split rates underestimates distribution drift (worker composition, seasonal fluctuation), so post-launch recalibration is mandatory.

Where it stops holding

Tiering presumes predictable difficulty. For some tasks difficulty is individual rather than item-level — hard for one worker, easy for another (professional-background differences) — where item-level prediction fails and only per-worker reliability routing works. Extremely hard tasks (single-worker accuracy near chance) have no solution inside the redundancy framework: the required copies diverge, and the right answer is redesigning the task to be easier (better instructions and examples) or switching to an expert channel, not stacking redundancy. Tier adjustment also lags — by the time a tier proves insufficient, dispatched tasks cannot regain redundancy except by after-the-fact re-queueing, which costs more than provisioning correctly at launch.

Applying it

• Maintain a difficulty-to-copies lookup table calibrated on small traffic: per task type, run a gold subset, measure single-worker accuracy, and set launch redundancy from the table.
• Route dynamically within batches: preset a cascade of "two copies first, automatic third on split," letting redundancy follow disagreement rather than spreading evenly.
• Flag contested-band items (historically high-disagreement features) into the high-redundancy tier; clear-band items get one copy plus spot audits.
• Re-calibrate the table quarterly with gold replays before distribution drift mis-tiers the system.
• Verification: track each tier's post-split re-queue rate and final rework rate; a persistently high re-queue rate means the tier launched under-provisioned, while near-zero re-queue with high cost share means over-provisioning — step down a tier.

Related

• Same group: V9.05.1 Having several people repeat the same task and comparing answers is the base quality control · V9.05.3 Seeded gold questions continuously estimate contributor accuracy · V9.05.4 Agreement does not guarantee correctness; shared misconceptions err in unison · V9.05.5 Paying per completed item induces fast, low-quality work
• Nearby: V9.04 Crowdsourcing Task Decomposition and Instructions · V9.06 Contributor Motivation and Payment
• Search terms: redundancy level · cost-accuracy tradeoff · task difficulty