Natural Expression of a Machine Learning Model's Uncertainty Through Verbal and Non-Verbal Behavior of Intelligent Virtual Agents

Video

DOI

Uncertainty cues are inherent in natural human interaction, as they signal to communication partners how much they can rely on conveyed information. Humans subconsciously provide such signals both verbally (e.g., through expressions such as "maybe" or "I think") and non-verbally (e.g., by diverting their gaze). In contrast, artificial intelligence (AI)-based services and machine learning (ML) models such as ChatGPT usually do not disclose the reliability of answers to their users. In this paper, we explore the potential of combining ML models as powerful information sources with human means of expressing uncertainty to contextualize the information. We present a comprehensive pipeline that comprises (1) the human-centered collection of (non-)verbal uncertainty cues, (2) the transfer of cues to virtual agent videos, (3) the annotation of videos for perceived uncertainty, and (4) the subsequent training of a custom ML model that can generate uncertainty cues in virtual agent behavior. In a final step (5), the trained ML model is evaluated in terms of both fidelity and generalizability of the generated (non-)verbal uncertainty behavior.