Closing the Loop: User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System

Human-in-the-loop topic modeling allows end users to guide the creation of topic models and improve the models' quality without requiring expertise in topic modeling algorithms. Prior work in this area either focuses on refinement implementation without understanding how users actually wish to improve the model or focuses on user wants without exposing them to a the effect user input has on the model. This work implements a set of user-preferred refinements identified from prior work. An interview study with twelve non-expert participants examines how end users are affected by issues that arise when the user is truly brought into the loop of the algorithm process. As these issues mirror those identified in interactive machine learning more broadly, such as unpredictability, latency, and trust, this work provides a mechanism for examining interactive machine learning challenges with non-expert end users through the lens of human-in-the-loop topic modeling. We find that although users experience unpredictability, their reactions vary from positive to negative, and surprisingly, we do not find any cases of distrust, but instead note instances where users perhaps trust the system too much or have too little confidence in themselves.