BackSwipe: Back-of-device Word-Gesture Interaction on Smartphones

Document Title

BackSwipe: Back-of-device Word-Gesture Interaction on Smartphones

Document Information

• Subject Area: Smartphone interaction design and human-computer interaction
• Keywords: word input, command input, gesture input, touchscreen, smartphone, user evaluation, interface design, one-handed interaction, back-of-device interaction

Research Background and Issues

• Identified Problems or Challenges:

  • Smartphone touchscreen interaction suffers from the "fat-finger problem" and limited reachable screen areas, affecting the convenience of one-handed use.
  • Existing back-of-device interaction methods are limited to simple gestures (e.g., swiping or tapping) and fail to fully explore the potential of this interaction mode.
  • Back-of-device input is constrained by hand posture limitations and visual occlusion issues, resulting in lower gesture positioning accuracy.

• Significance:

  • With the development of smart devices featuring back-of-device displays (e.g., Samsung foldable phones), back-of-device interaction technology is becoming increasingly practical. Expanding the capabilities of back-of-device interaction, particularly for word and command input, can enhance smartphone functionality and improve user experience.

• Research Motivation and Related Work:

  • Word-gesture typing technology has been widely applied in front-screen interaction, laying a foundation for implementing gesture input on the back screen.
  • Related studies have explored blind input, virtual keyboard layout inference, and user-defined gestures, but have not provided systematic solutions for full-function word input on the back screen.

Solution

• Proposed Method or Solution:

  • A novel input method called "BackSwipe" is proposed, enabling users to input words and commands through back-of-device gestures, supporting one-handed use.
  • A back-of-device word gesture decoding algorithm is designed and implemented, capable of automatically inferring the position and size of the virtual keyboard and dynamically adjusting to fit the user's gesture scale.

• Innovations:

  • Introduced a dynamic algorithm for automatic inference of virtual keyboard layout, combined with word gesture decoding technology to support back-of-device interaction.
  • Addressed accuracy issues in back-of-device input and provided a novel input space and interaction method.
  • BackSwipe avoids visual and interaction interference on the front screen, offering a dedicated back-of-device input method.

• Implementation Steps and Key Technologies:

  • Virtual Keyboard Position Inference: Using user input gestures, the Dynamic Time Warping (DTW) algorithm is employed to infer the center position and key distribution of the virtual keyboard.
  • Word Gesture Decoding: A language model (GPT pre-trained model) is combined with gesture data for decoding, supporting continuous word input and command triggering.
  • Interaction Design: Real-time suggestions and switching mechanisms are provided to help users correct errors and improve input efficiency.

Research Outcomes

• Specific Results:

  • BackSwipe achieved an average command input accuracy of 92% and an average input speed of 5.32 seconds per command.
  • In text input tasks, users achieved an average input speed of 9.58 WPM, with some users reaching up to 18.83 WPM; the average word error rate was 11.04%, with the lowest rate at 2.85% for some users.

• Advantages Over Existing Solutions:

  • BackSwipe avoids front-screen gesture conflict issues, allowing users to input while viewing content or engaging in other interactions.
  • Provides a more natural and faster input method for one-handed use, particularly suitable for short text or command input.

• Experimental or Evaluation Results:

  • User Experiment 1: Verified that users could draw continuous word gestures on the back screen and accurately recall the Qwerty keyboard layout. Input speed averaged 21.9 WPM.
  • User Experiment 2 (Command Input):
    • Average command input time: 5.32 seconds
    • Average error rate: 7.5%
  • User Experiment 3 (Text Input):
    • Average input speed: 9.58 WPM
    • Average error rate: 11.04%
    • User feedback: Moderate physical and mental workload (average scores of 5.7 and 4.9, respectively).

• Limitations and Future Directions:

  • Limitations: Prolonged use of back-of-device input may cause fatigue, making it suitable for short text or command input rather than long text tasks.
  • Future Directions:
    • Further optimize decoding algorithms to improve speed and accuracy.
    • Explore the multifunctionality of back-of-device interaction, such as combining simple gestures with text input.
    • Adapt to different handheld devices or screen designs to enhance cross-device compatibility.