Every data field needs a stated purpose
Aliases: data inventory · field-level purpose · data dictionary
What it is
Field-level purpose specification assigns a testable functional purpose to every datum entering a system instead of covering a table with phrases such as “improve experience” or “business needs.” The specification links a field to processing, readers, and an output or decision, making it possible to ask whether the field is necessary, replaceable, and still justified after its use ends.
Why it happens
Dataset-level governance lets new columns inherit broad permissions and vague purposes, eventually producing orphaned fields nobody can explain. Field-level mappings expose hidden dependencies and give access, deletion, and user communication a common semantic unit. They also create accountability: a new use appears as a mismatch in the mapping rather than silently inheriting an old authorization.
Studying it
Researchers can reconcile catalogs, query logs, product requirements, and code paths, sampling field owners to explain how an input changes a concrete output. Measures include ownerless fields, agreement between stated and observed use, stale unread fields, cross-purpose queries, and inter-rater agreement about purpose. A plausible interview account is not evidence of actual use; lineage and runtime logs are needed, while their own incompleteness must be documented.
Where it stops holding
Fields and purposes are not always one-to-one. Free text may contain unexpected attributes, and raw sensor streams permit inferences not represented in the schema. Byte-level descriptions create false precision; the appropriate unit should follow access, risk, and deletion boundaries. Exploratory or security research can retain wider hypotheses, but still needs a bounded environment, expiry, and prohibited uses rather than an indefinite “research” exemption.
Applying it
- Require definition, source, functional purpose, readers, output, sensitivity, and review date when registering a schema field.
- Replace labels such as “analytics” or “personalization” with observable product actions and note acceptable substitute inputs.
- Link new queries and disclosures to the catalog automatically; route unmatched uses to review rather than inheriting permission.
- Quarterly, trace sampled interface actions to queries and fields back to functions; remove ownerless, unread, or unjustified fields and run dependency tests afterward.