Visible data is not verifiable data — provenance matters too

What it is

Publishing a processed chart or a headline number, and providing verifiable raw data, are different things. Verifiable means a third party can redo the analysis: obtain the data at original granularity, know the definitions and processing steps, and check whether the conclusion is what the data necessarily yields. The metadata that supports this is data provenance — where the data came from, what transformations it passed through, what choices were made at each step.

When only finished products circulate, "transparency" has not actually occurred: what the public sees is still an assertion, merely drawn as a chart.

Why it happens

Why is a finished chart not verifiable? Because between raw data and product, every processing step embeds a degree of freedom: which statistical definition (registered or resident population), what aggregation level (district or block), how periods are cut (calendar or fiscal year), how outliers are cleaned, which months are excluded. Each choice can flip the conclusion's direction, and none is visible in the finished chart.

Publishing only conclusions publishes the outcome of one set of choices while hiding the choosing. An auditor cannot distinguish "the data really supports this" from "a different, equally defensible definition reverses it". Open-data research lists definitional opacity and quality among the top barriers precisely for this reason: an analysis that cannot be redone does not exist, evidentially.

Studying it

• Open-data quality and provenance studies: systematic assessments of portals' metadata completeness (definitions, collection method, revision records), examining how provenance information relates to third-party reuse.
• Data journalism practice research: following newsroom practices of reproducible analysis — publishing processing scripts and raw data alongside stories so readers can recompute.
• Re-analysis comparisons: independent teams redo a published analysis from the open raw data and compare conclusions; consistency is then related to the completeness of provenance.

One methodological caution: measure verifiability by task completion, not by material presence — not "how many metadata fields are filled in", but whether an external team can actually reproduce the number from public materials. Where reproduction fails is where the provenance gap sits.

Where it stops holding

• Privacy limits raw granularity. Individual-level records cannot be released, but aggregating to district level can erase small groups. Granularity is a weighed decision and should be stated — a vague "withheld for privacy" is routinely used as a shield for not publishing.
• Cost and security exceptions invite abuse. Commercial contracts and security details genuinely warrant exceptions, but exceptions should be itemised and reviewable; blanket clauses amount to a licence to withhold.
• Raw data cannot certify its own quality. Provenance is filled in by the publishing party and can be fabricated; cross-institutional corroboration — two independent collection sources agreeing — is the harder constraint.

Applying it

• Attach a metadata file to every released dataset: collection method, definitions, known defects, update cadence, responsible department.
• Provide the processing trail from raw to product — ideally the scripts, at minimum a step-by-step record of choices; finished charts link to the exact dataset version used.
• Version the datasets: updates leave traces and historical versions remain retrievable — analyses done on an older version should still reproduce after the data moves on.
• How to check: take a published headline conclusion and have an external team reproduce its key numbers from public materials alone. Whatever fails to reproduce gets provenance information added, item by item, until it does.

Related

• Same group: Z8.04.1 Publishing city data helps the public examine the basis of governance decisions · Z8.04.2 Technical barriers in how data is presented limit actual public participation · Z8.04.3 Participation channels need closure, with visible follow-through on feedback
• Nearby: Z8.03 Noticing public sensing and surveillance
• Search terms: data provenance · open data quality · reproducible analysis · open government data