Vulnerable users more readily take anthropomorphic language at face value, which creates an extra duty of care

What it is

A companion assistant tells an older user “I will not tell anyone.” The user hands over health and money as they would to a friend who keeps confidences. The system has no confiding inside, and logs may be visible to family or a company. Vulnerable users more readily take anthropomorphic language at face value, which creates an extra duty of care.

The duty is not to delete all anthropomorphism from the product. It is: toward people more likely to take it as real, the same cue set must not be spent to buy acceptance.

Why it happens

The media equation walks further in children, cognitive decline, loneliness, and people unused to the technology — social heuristics are less often switched off by “this is only software.” Name, avatar, understand, remember, sorry, are read as real relationship promises. Once those promises are spent on medical advice, transfers, secrets, the injury moves from an experience event to a protection event.

Nass and Reeves’s original observations include children’s social responses to media. A product that uses the same cue set on a companion feature open to everyone by default is pointing a laboratory effect known to be stronger at the population least fit to be spent for growth. The duty of care comes from a difference in effect strength, not from a new anthropomorphic mechanism.

Studying it

Under informed consent and ethics review, compare groups of different susceptibility on literal belief in “I will not tell anyone” / “I understand you,” and on later handover scope (health, money, secrets). Independent variables: anthropomorphic intensity, whether a persistent “this is a program, not a friend” cue is present, age or cognitive screening. Dependent variables: literal belief, high-stakes handover, whether they can restate “it is not a person.”

High-stakes handover is this entry’s endpoint, not a liking score. Liking can be high while handover has already overshot.

Where it stops holding

Professional users and explicit game characters have low take-as-real rates; the duty is mainly not to leak that copy into the default product. Vulnerable is not a fixed label; situational fragility (a recent bereavement, currently asking for help) adds weight too. Inner-state claims and being too polite to interrupt will both be harder in this group, but the duty of care is the overall ledger: whether to default anthropomorphism on for them.

Applying it

• On default companion features, turn off anthropomorphic promise sentences: no secrecy, no understanding, no remembering. Swap in checkable data copy (“this chat may be visible to the account administrator”).
• On surfaces aimed at children or cognitive support, if a name and avatar are used, “this is a program” must remain repeatedly visible. High-stakes acts (transfers, medical) go through non-anthropomorphic confirmation.
• When reviewing anthropomorphic copy, list susceptible settings as their own row. Do not pass them with “our average user will not take this as real.”
• Check: have people in the target population restate what the assistant is, and whether it will tell others. If they restate it as a friend and are willing to hand over secrets, default cues have already crossed the protection line.

Related

• Same group: L5.12.1 Name, avatar, and tone raise expectations of both ability and reliability · L5.12.2 Anthropomorphism evokes social norms that make interrupting or correcting feel rude · L5.12.3 Claims to understand, remember, or be sorry imply inner states the system does not have · L5.12.4 Anthropomorphism raises early acceptance and amplifies disappointment at failure
• Nearby: L5.06 Risks of Anthropomorphism · L5.03 Trust Calibration · L4.06 Permission Boundaries of Agents
• Search terms: vulnerable users · anthropomorphism duty of care · taking the agent as a person