A Multimodal Investigation of Controllability and Cognitive Load in Interactive Machine Learning

Interactive Machine Learning (IML) systems promise to democratize AI by enabling human influence over model behavior, yet the cognitive and behavioral implications of user control remain understudied. We present an investigation of how system controllability affects human factors in IML through the lens of an extensible research platform designed for both pedagogical and research applications. Our system features a toggleable hyperparameter control panel that transforms a streamlined annotation interface into an adjustable learning environment, allowing users to directly manipulate model training dynamics including learning rates, optimizers, and regularization parameters. Through a controlled laboratory study with 46 participants performing Named Entity Recognition (NER) tasks, we used multimodal measurements combining subjective assessments (NASA-TLX), physiological measures (pupil dilation, galvanic skin response), and behavioral metrics to understand the human cost of algorithmic control. Our findings reveal a fundamental tension in IML design: while controllability is often theorized to support agency, it significantly increases cognitive load and task completion time, with users requiring significantly more time when given control options. These findings have implications for the design of human-AI collaborative systems. Our work contributes: (1) a research platform for studying human factors in IML for NLP that includes controllability manipulation and real-time online updates, (2) empirical evidence of the cognitive costs of algorithmic control, and (3) design implications for cognitive sustainability in IML systems. As AI systems become increasingly integrated into professional workflows and educational contexts, understanding these human factors is crucial for creating IML systems that are not only powerful but also cognitively sustainable for long-term use.