Evaluate independently before aggregating; prior discussion erases independence and distorts the number effect

What it is

A multi-evaluator process should have each person inspect, record, and initially rate problems before unified deduplication and discussion. If the group meets first or shares preliminary findings, later evaluators are anchored: overlap is artificially increased and added evaluators appear more productive even though independent coverage has not grown.

Why it happens

Verbal findings, shared documents, and group chat leak attention direction. Discussion first draws evaluators toward known issues and reduces exploration of their own paths; wording may also merge distinct defects. Independent initial review preserves each person’s search strategy and interpretation, allowing the team to measure complementarity and real coverage.

Where it stops holding

Independence does not forbid common training. Evaluators may share the task script, problem template, and checklist interpretation, but not findings. An urgent review can run quickly in parallel and aggregate immediately; process completeness should not delay fixing a hazard.

Applying it

• Have each evaluator record problems, locations, evidence, and initial severity in a private document or session.
• After all submissions, a coordinator deduplicates, merges, and creates a matrix before the discussion.
• In discussion, confirm facts and severity first, then add tags and repair directions; record rejected findings.
• Compare additions and deletions between the independent and discussion stages to detect excessive anchoring.

Related

• Same group: B3.19.1 A single evaluator finds only a small part of all problems, and individuals differ widely · B3.19.2 Total findings show diminishing returns as evaluators increase, often flattening after three to five people · B3.19.3 Diminishing-return estimates assume independent evaluators with equal detection probability, conditions rarely met · B3.19.4 Low overlap means the problem space is large and more evaluators are needed, not that quality is poor
• Nearby: Q2 Usability Evaluation · Q4 Research Methods and Evaluation
• Search terms: independent evaluation · aggregation bias · anchoring effect