Privacy-sensitive tests need de-identified data, not live production feeds

What it is

Wanting real edges is not the same as wiring the production store to a lab laptop. When identity, health, money, or message content is involved, the right material is de-identified (anonymized) fixtures: keep length, missingness, duplicates, and odd formats; strip identifiable people. A live production feed turns evaluation into an unauthorized extra copy—participants, observers, recordings, and shared drives all become new replicas. Realism wants distribution and shape, not a particular customer’s name.

Why it happens

Production data is dangerous not because it is “too real,” but because copies spread: test accounts, hosted prototypes, recordings, and chat screenshots will not live under production access control. De-identification that only “changes the name to Jane” still leaves address, rare disease, and unique amounts re-identifiable. Effective de-identification targets that risk: generalize, drop direct identifiers, cut joinable keys, while deliberately keeping the statistical shape that layout cares about. Teams often swing between poles—all Lorem, or “just connect production, it’s easier”—and skip the fixture work in between. Once built, fixtures can be versioned and are more repeatable than a live pull.

Studying it

Write the data source into ethics and method: sampling rule from production, de-identification procedure, re-identification risk assessment, who may touch the unidentified source. A check: do known layout failures still reproduce on the de-identified set? If they all vanish, the tail was washed out too. k-anonymity-type bounds can be a floor; UI tests care more that field-level shape remains. Archive recordings and materials under minimum-necessary rules; delete unidentified caches after the session. “Staff signed an NDA” is not a substitute—internal leaks are common too.

Where it stops holding

Some defects fire only on a particular live row (a corrupt legacy encoding); de-identification may delete it. Reproduce that in a controlled setting, with authorized people on a production read-only copy—not by handing the store to a usability lab. Synthetic data can be very true in shape and still false in semantics. Legal or security audits may forbid any derived copy; switch to read-only screen share and no local download. When the participant is the data subject, using their own data is a different consent, and observer scope must still be limited.

Applying it

• Default: no production credentials in the prototype environment; when you need live shape, use a de-identified fixture repo.
• Write a field-level de-identification list: drop direct identifiers, generalize quasi-identifiers, keep length and missingness.
• Before testing, check that the fixtures still contain empty, long, and duplicate rows; if the tail is gone, rebuild fixtures rather than hot-wiring production.
• Before recording, confirm the frame shows no identifier that should not be there; treat a leak as an incident and stop.

Related

• Same group: Q5.11.1 Placeholder data hides edge cases that real data would surface · Q5.11.2 Extreme length and malformed data belong in prototype tests · Q5.11.4 Attractive sample data leads reviewers to underestimate live complexity
• Adjacent: Q1.06 Research ethics · Q5.06 Canary release and pilots
• Search terms: de-identified prototype data · anonymization · re-identification risk