Extending Keyboard Shortcuts with Arm and Wrist Rotation Gestures

We propose and evaluate a novel interaction technique to enhance physical keyboard shortcuts with arm and wrist rotation gestures, performed during keypresses: rolling the wrist, rotating the arm/wrist, and lifting it. This extends the set of shortcuts from key combinations (e.g. ctrl + v) to combinations of key(s) and gesture (e.g. v + roll left) and enables continuous control. We implement this approach for isolated single keypresses, using inertial sensors of a smartwatch. We investigate key aspects in three studies: 1) rotation flexibility per keystroke finger, 2) rotation control, and 3) user-defined gesture shortcuts. As a use case, we employ our technique in a painting application and assess user experience. Overall, results show that arm and wrist rotations during keystrokes can be used for interaction, yet challenges remain for integration into practical applications. We discuss recommendations for applications and ideas for future research.