ChoiceMates: Supporting Unfamiliar Online Decision-Making with Multi-Agent Conversational Interactions

From purchasing a gift to deciding on a hobby, unfamiliar decisions---decisions without domain knowledge and experience---are frequent and significant. The complexity and uncertainty of such decisions demand unique approaches to information seeking, understanding, and decision-making. Our formative study highlights that in the current workflow, users want to start by discovering broad and relevant domain information evenly and simultaneously, quickly address emerging inquiries, and gain personalized standards to assess information found. We present ChoiceMates, an interactive multi-agent system designed to address these needs by enabling users to engage with a dynamic set of LLM agents each presenting a unique experience in the domain. Unlike existing multi-agent systems that automate tasks with agents, the user orchestrates agents to assist their decision-making process in each turn, through chatting with all agents, with a tagged subset of agents, or calling in new agents into the space. By comparing ChoiceMates with a web search condition and a multi-agent framework (n=12), we show that ChoiceMates enables a more confident, satisfactory decision-making with better situation understanding than web search, and higher decision quality than a commercial multi-agent framework. We further illustrate how participants utilized ChoiceMates to make unfamiliar decisions, providing insights into designing a more controllable and collaborative multi-agent system.