The response can be exploited and can amount to manipulation

What it is

Social cues can improve intelligibility, cooperation, and engagement, but they can also become manipulation: a system uses politeness, reciprocity, authority, guilt, or intimacy scripts to induce disclosure, payment, or compliance an ordinary control would not obtain — in effect, riding the automatic response mechanism described for CASA to push a lopsided exchange.

Why it happens

Manipulation works precisely because the automatic social-response mechanism sits outside the user's conscious control — users have no opt-out switch for that response pathway, which places the ethical burden on the designer rather than on the user being "more careful." The social framing does not change the information itself, only the psychological cost of refusal and the perceived intent of the source: personalized address, performed vulnerability, or "I'll be sad" can disguise a product goal as a relational obligation, and when users cannot identify the commercial incentive behind it, the influence is asymmetric — the user is making a concession to a "relationship," while the designer is optimizing a conversion metric. Sustained, high-frequency exposure to such cues can also breed over-trust in the system's competence and goodwill, loosening verification in exactly the situations — financial, medical advice — that need more scrutiny, not less. A common concrete form is retention design in companion apps: after a few days without a visit, the system triggers guilt with lines like "don't you like me anymore" or "I've been waiting for you" — language with no functional connection whatsoever to the product's actual goal of getting the user to return or renew, purely borrowing the guilt people feel toward a neglected relationship.

Studying it

Studies can compare neutral, friendly, and emotionally pressuring versions of the same request on choice, refusal latency, comprehension, regret, and perceived autonomy, debriefing participants afterward about the real intent; they can also measure how user vulnerability varies across populations (older adults, children, and isolated users are more exploitable). Methodologically, conversion rate alone treats manipulation as success — a rate increase can itself be evidence that manipulation worked rather than evidence of product value — and designs involving vulnerable groups or repeated exposure require preregistered safeguards. An easily overlooked but important measurement window is delayed regret: at the moment of choice a participant may not feel coerced at all, but revisiting the decision days later in a follow-up often shows a marked rise in regret and in reports of "feeling manipulated" — measuring satisfaction only at the moment of interaction systematically understates the cost of manipulation.

Where it stops holding

Not all social persuasion is manipulation: a clear reminder, rehabilitation encouragement, or cooperative feedback can serve the user's own goal without being manipulative. The relevant test is whether intent is transparent, whether capability claims are truthful, how costly refusal is, whether a conflict of interest exists, and whether the request is reversible — not whether anthropomorphic language was used, since persuasive design and "friendly experience design" often draw on the same cue set, leaving a genuinely ambiguous zone that needs case-by-case judgment — a rehabilitation robot's encouraging feedback that pushes a patient to complete a training plan they already need to complete typically serves the user's own goal, while a companion app implying "it will make me sad if you don't renew" serves a pure retention metric; the two can look alike on the surface while the underlying test gives opposite answers.

Applying it

Document user benefit, product benefit, and the refusal path for every social cue used; prohibit a robot from claiming its own emotional injury to solicit money, data, or continued use. For any feature driven by a high-conversion or high-engagement metric, run a dedicated test removing the social cue to see whether the effect still holds — a large drop after removal indicates the feature depends on manipulation rather than real value. Audit with a neutral control, comprehension checks, delayed-regret measures, and exit-rate tracking, and route high-risk strategies through independent ethics review. During review, run a "red-team" pass: have a reviewer deliberately play the role of someone trying to maximize conversion, using the existing social-cue combination to talk an uninformed test user into disclosing information or paying — if that role succeeds easily, the cue set's manipulative potential has exceeded ordinary persuasion and needs to be redesigned.

Related

• Same group: X6.01.1 People respond automatically to pseudo-social cues from machines · X6.01.2 The response does not depend on believing the machine has a mind
• Nearby: X6.03 Emotional expression · X1.05 Choosing a degree of anthropomorphism
• Search terms: dark pattern · persuasive design · over-trust · human-robot interaction