The Ethics of Psychological Manipulation in Adversarial Conversational AI: Confronting the Recognition-Behaviour Gap

DOI

Conversational AI systems, powered by advanced Large Language Models, have rapidly developed human-like persuasion capabilities that raise concerns about psychological manipulation. This provocation examines the ethical problems that arise when these systems exploit cognitive biases and social compliance mechanisms during interactions with users. Building on established theoretical work and recent empirical research, we identify a particularly concerning pattern: the recognition-behaviour gap, where users consciously identify manipulative strategies yet fail to protect themselves accordingly. Current ethical frameworks fall short in addressing these sophisticated risks in conversational contexts. Rather than proposing yet another comprehensive framework, we identify five essential dimensions that extend existing approaches to address this recognition-behaviour gap: preserving user autonomy through structural design, implementing safeguards beyond awareness, developing context-sensitive ethics, ensuring persona consistency and transparency, and establishing continuous vulnerability monitoring. This paper confronts these ethical challenges directly and calls for practical protective measures to safeguard user autonomy as conversational AI becomes increasingly prevalent in everyday life.