Scaling Context-Aware Task Assistants that Learn from Demonstration and Adapt through Mixed-Initiative Dialogue

DOI

Daily tasks such as cooking, machine operation, and medical self-care often require context-aware assistance, yet existing systems are hard to scale due to high training costs and unpredictable and imperfect performance. This work introduces the PrISM framework, which streamlines the process of creating an assistant for users' own tasks using demonstration and dialogue. First, our tracking algorithm effectively learns sensor representation for steps in procedures from a single demonstration. Second, and critically, to tackle the challenges of sensing imperfections and unpredictable user behaviors, we implement a dialogue-based context adaptation mechanism. The dialogue refines the system's understanding in real time, thereby reducing errors such as inappropriate responses to user queries. Evaluated through multiple studies involving several examples of daily tasks in a user's life, our approach demonstrates improved step-tracking accuracy, enhanced user interaction, and an improved sense of collaboration. These results promise a scalable, multimodal, context-aware assistant that effectively bridges the gap between human guidance and adaptive support in diverse real-world applications.