What Could Possibly Go Wrong When Interacting with Proactive Smart Speakers? A Case Study Using an ESM Application

Title of the Paper

What Could Possibly Go Wrong When Interacting with Proactive Smart Speakers? A Case Study Using an ESM Application

Paper Information

• Research Domain: Voice User Interfaces (VUIs) and Human-Computer Interaction with Smart Devices
• Keywords: Voice User Interface, Smart Speaker, Voice Assistant, Google Home, User Experience, Interaction Errors

Research Background and Problem Statement

• Identified Issues or Challenges: Interaction errors in Voice User Interfaces (VUIs) occur frequently, potentially affecting user experience and the practicality of applications. Particularly, proactive VUIs may introduce additional complexities during interaction.
• Importance of the Problem: Proactive smart speakers are considered to have significant potential in areas such as data collection, healthcare, and entertainment. However, errors encountered during actual use may lead users to abandon these devices.
• Research Motivation and Related Work: Existing studies primarily focus on VUIs in laboratory settings or summarize common errors in standard smart speakers. There is a lack of detailed analysis of interaction errors in proactive smart speakers within real-world environments.

Proposed Solution

• Proposed Solution: To study interaction errors in proactive smart speakers, the authors developed a proactive speaker prototype based on Google Home and designed a voice-driven Experience Sampling Method (ESM) application.
• Innovative Aspects of the Solution:
  • Introduced proactive speakers that initiate interactions by asking users questions and collecting data, rather than merely responding to user commands.
  • Combined audio recording and interaction logs to identify interaction errors and analyze how users respond to these errors.
• Implementation Steps and Techniques:
  1. Developed a hardware prototype integrating Raspberry Pi and Google Home to enable "proactive" triggering functionality.
  2. Created custom Google Actions to implement multi-turn voice interactions for experience sampling content.
  3. Deployed the prototype in the homes of 13 participants for three weeks to collect data.
  4. Analyzed the occurrence rate of errors, user strategies, and interaction time costs using audio recordings and interaction logs.

Research Findings

• Specific Findings:
  • A total of 1,213 ESM data points were collected, including complete voice interaction audio recordings and 30,000 interaction logs.
  • Even for simple tasks like answering numerical questions, proactive smart speakers using Google Home exhibited significant interaction error frequencies. These included default system errors (e.g., "no input" or "mismatch" errors) and unexpected termination errors (e.g., erroneous triggering of map search functionality).
  • Users employed various strategies to address errors, such as increasing volume, repeating responses, changing vocabulary/pronunciation, and seeking assistance from others.
• Advantages Over Existing Solutions:
  • Conducted a comprehensive analysis of proactive smart speakers operating in real-world home environments, rather than limiting the study to laboratory testing.
  • Provided detailed classifications of user strategies for addressing proactive speaker errors, offering practical guidance for future design.
• Experimental or Evaluation Results:
  • 37.2% of complete interactions experienced at least one system error, while 14.4% of records were incomplete due to interaction interruption errors.
  • Errors significantly increased the time cost for users to complete tasks, negatively impacting the usability of proactive speakers.
• Limitations and Future Directions:
  • The study was limited to the Google Home platform, and findings may not be generalizable to all smart speakers or VUI applications.
  • Privacy concerns related to audio recordings need further resolution.
  • Future work could expand proactive speakers to support more complex interactions, explore broader user groups, and utilize auxiliary sensors (e.g., cameras) to study the impact of environmental interference on interactions.

Conclusion

This paper conducted a long-term real-world experiment to investigate interaction errors in proactive smart speakers and their impact on user experience. It also proposed specific recommendations for optimizing voice-driven VUI application design and research methodologies. These findings contribute to improving the practicality of smart speakers and lay a foundation for future VUI application development.