Agreement needs a shared codebook and explicit definitions

What it is

Two people applying similar codes to the same material only yields intercoder agreement when they share a written rule for inclusion, exclusion, and positive and negative examples. “We see it the same way” without a manual is a temporary truce: each may be using a private boundary, the numbers look aligned, and the next batch will scatter. Codebook-supported agreement moves the repeatable object from “these two coders” to “this set of definitions.”

Why it happens

Categories are not ready-made ticks on the material. Coders decide where a stretch starts and ends, which cell it falls into, and whether it may fall into two. If those decisions live only in memory, apparent agreement may come from shared disciplinary instinct or from talking it out in the room, not from a rule that can be handed on. The codebook writes the decisions as checkable sentences: name, definition, boundary cases, contrast with neighboring codes, unit of analysis. Once the unit is vague—whole paragraph, turn, or semantic claim—even a clear category definition will have its agreement eaten by segmentation disagreements.

Studying it

Freeze a codebook version before independent coding starts, and trial-code a small set the authors did not use while writing it. Report unitizing agreement and category agreement separately; do not hide both in one overall rate. Recode systematic disagreements back into the manual: overlapping definitions, missing negative examples, or an unstated unit rule. Later agreement should be measured on new material, not on the batch just used to revise the manual—that only shows the coders remember the discussion.

Where it stops holding

Early open coding is inventing categories; demanding a complete manual at that stage freezes exploration. Once themes will be reported or work will be split across people, an agreement number with no definitions has no interpretive force. An over-fine manual can also turn coding into mechanical form-filling and disguise interpretive judgment as ticking boxes. Some studies keep an “other / uncertain” code; it needs a rule for when it may be used, or it becomes a bin for every disagreement.

Related

• Same group: Q4.11.2 Low agreement coefficients diagnose ambiguity in the coding frame · Q4.11.3 Reaching agreement can surface neglected categories · Q4.11.4 High agreement is not analytic value
• Adjacent: Q4.02 Coding and thematic analysis · Q1.07 Researcher bias
• Search terms: codebook · intercoder agreement · unitizing