Multiple coders require an agreement check
Aliases: multi-coder agreement · independent double coding · unchecked consensual coding
What it is
Two people “code together,” nodding at one screen, and hand over a shared list. That shows they were willing to agree in the room. It does not show that the same stretch would be marked the same way if they read it apart. Intercoder agreement check means at least some material is coded independently, then compared. Extra people do not by themselves add rigor. Uncompared conferral just replaces one person’s judgment with the mood of a room.
Why it happens
Coding draws a boundary on fuzzy language. Where the boundary sits slides with reading order, fatigue, and tacit ideas about what the project is trying to show. Coding in company lets the first speaker set the boundary; later speakers repair the social relation, not the extract. Independent coding exposes each person’s boundary: the same “it’s fine” is satisfaction, reluctance, or uncodable. Comparison is not a hunt for the right person. It asks whether the material is stable under the current codes. Without a check, every later “our coding found…” assumes coders are interchangeable. That assumption is usually false.
Studying it
Before live coding, reserve an overlap sample and forbid meeting until independent passes are done. Log item-level matches and mismatches, which codes collect the disputes, and whether definitions move after comparison. A write-up should state the overlap fraction and the stage at which comparison happened; without independent comparison, “two-person coding” is not a quality claim. What is compared is whether judgments coincide, not whose title is higher.
Where it stops holding
Exploratory, single-analyst close reading aimed at generation rather than a stable classification does not need a second coder for form’s sake; call it a single reading instead of pretending it was multi-checked. Two people coding different interviews with no overlap cannot produce a check. Near-transcription marks such as timestamps or speaker IDs often agree highly and cannot stand in for comparison on interpretive codes. Agreement between a model and a person is a different check; internal model stability is not interpersonal agreement.
Applying it
- If findings will be issued as “coding results,” reserve an independent overlap stretch: code apart, then compare.
- Do not look at each other’s codes before comparison; the meeting handles disputed items only, and does not re-watch the matches to manufacture reassurance.
- In the method note, state whether coding was independent, how much material overlapped, and what happened after comparison. If those three cannot be written, do not write “coded by multiple people.”
- If there is no time for a check, downgrade the claim to “one analyst’s reading” and mark that no comparison was done.