HMInference: Inferring Multimodal HMI Interactions in Automotive Screens

Video

Driving requires high cognitive capabilities in which drivers need to be able to focus on first-level driving tasks. However, each interaction with the User Interface (UI) system presents a potential distraction. Designing UIs based on insights from field-collected user interaction logs, as well as real-time estimation of the most probable interaction modality, can contribute to engineering focus-supporting UIs. However, the question arises of how user interactions can be predicted in in-the-wild driving scenarios. In this paper, we present HMInference, an automotive machine-learning framework which exploits user interaction log data. HMInference analyzes the interaction sequences of users based on UI domains (e.g., navigation, media, settings) and driving context (e.g.,vehicle trajectory) to predict different interaction modalities (e.g., touch, speech). In 10-fold cross-validation, HMInference achieves a mean accuracy of 73.2% (SD:0.02). Our work advances areas where user interaction prediction for in-car scenarios is required e.g., to enable adaptive system designs.