Building Appropriate Mental Models: What Users Know and Want to Know about an Agentic AI Chatbot

DOI

Agentic systems aim to handle complex problems with increasing system autonomy using generative AI. These new agentic systems are becoming more feasible and easier to build. Yet we know little about what end-users need to know to use these systems appropriately. We study one such agentic system, "Gent," which can break down complex problems into a set of actions, provide a rationale for each action, interact with external information, and cite its sources. Our goals were to understand users' mental models of the agentic system, the information users leveraged to evaluate the accuracy of the system, and users' information needs. In our study (N=24), participants interacted with Gent for four information seeking tasks where they could see Gent’s actions, rationale, and sources. Participants' mental models centered around the search-like qualities of the system, with their confidence impacted by the website sources. Participants' mental models often lacked insight into the workings of the generative AI model and agentic framework that impact the actions the system takes. Participants used the descriptions of the system's actions to support their evaluation of the accuracy of the system and wanted to know more about how the system got to its answers. Participants also relied on their own personal knowledge and the style or length of Gent's responses to evaluate the accuracy. Our results highlight the need for further transparency in agentic AI systems to support end-users in evaluating system outputs and help them build effective mental models.