In Whose Voice?: Examining AI Agent Representation of People in Social Interaction through Generative Speech

As generative artificial intelligence (genAI) applications gain popularity, there is a dearth of research examining how applications may transform social interactions. One possible application set to transform social interactions is the use of generative speech to power AI agents that can realistically represent people. Our work examines the potential implications of AI agents representing individuals in human conversations ("agent representation") as a way to begin filling this research gap. We take a multi-method approach, conducting formative interviews with developers, a co-design workshop with designers, a harm analysis among researchers, and interviews with the general public. Both technologists and potential users worry adopting agent representations might harm the quality, trust, and autonomy of human communication. Potential users are particularly concerned that agent representations could undermine the value of social interaction and threaten individuals’ ability to control their image. To avoid such potential consequences, future genAI-powered agents and speech applications should take into account user-defined red lines when considering applying these technologies in social settings.