Rewriting the Script: Adapting Text Instructions for Voice Interaction

Video

Voice assistants have sharply risen in popularity in recent years, but their use has been limited mostly to simple applications like music, hands-free search, or control of internet-of-things devices. They have also been critiqued for their weak conversation skills. What would it take for voice assistants to guide people through more complex tasks? In our work, we study the limitations of the dominant approach voice assistants take to complex task guidance: reading aloud written instructions. Using recipes as an example, we observe twelve participants cook at home with a state-of-the-art voice assistant. We learn that the current approach obscures the bigger picture and overwhelms users with too much information, among seven other challenges. Instructions delivered by a voice assistant are particularly difficult because they cannot be skimmed as easily as written instructions. We draw on our observations to propose eight ways in which voice assistants can "rewrite the script"---like summarizing, splitting, or reordering information---to transform written sources into forms that are readily communicated through spoken conversation. We conclude with a vision of how new work in natural language processing can be leveraged for intelligent systems to guide users effectively through complex tasks.