User-Driven Design Principles for Gesture Representations

Many recent studies have explored user-defined interactions for touch and gesture-based systems through end-user elicitation. While these studies have facilitated the user-end of the human-computer dialogue, the subsequent design of gesture representations to communicate gestures to the user vary in style and consistency. Our study explores how users interpret, enact, and refine gesture representations adapting techniques from recent elicitation studies. To inform our study design, we analyzed gesture representations from 30 elicitation papers and developed a taxonomy of design elements. We then conducted a partnered elicitation study with 30 participants producing 657 gesture representations accompanied by think-aloud data. We discuss design patterns and themes that emerged from our analysis, and supplement these findings with an in-depth look at users’ mental models when perceiving and enacting gesture representations. Finally, based on the results, we provide recommendations for practitioners in need of “visual language” guidelines to communicate possible user actions.