TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input

Document Title

TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input

Document Information

• Subject Area: Human-Computer Interaction; Sensor-Based Gesture Recognition; Multi-Task Learning
• Keywords: Gesture Recognition, Off-Screen Interaction, Accelerometer, Gyroscope, Multi-Task Learning, Mobile Sensors, TapNet, Dataset, Deep Learning

Research Background and Problem

• Problems or Challenges Identified by the Authors:

  • Current mobile device interaction primarily relies on touchscreens, which face limitations such as difficulty in one-handed operation and visual obstruction caused by the screen.
  • Few existing studies explore off-screen interaction, but most rely on additional hardware to achieve complex gesture recognition, increasing manufacturing costs and reducing portability.
  • Related work is often based on limited datasets and can only recognize single gesture attributes, with accuracy often insufficient.

• Importance of the Problem:

  • Off-screen interaction without additional hardware can expand the scope of user interface design, making interaction more convenient and free from visual obstruction.
  • Providing more accurate gesture recognition can enhance mobile phone usability, such as in scenarios where touchscreens fail to function properly, like when wearing gloves or underwater.

• Research Motivation and Related Work:

  • This study aims to improve technical models to establish a foundation for more efficient and versatile off-screen gesture recognition in practice.
  • The authors reference existing sensor-based gesture detection studies and highlight their inability to meet the demands for high-accuracy, multi-attribute recognition.

Solution

• Proposed Method or Solution:

  • The authors designed TapNet, a multi-task learning convolutional neural network (CNN) for recognizing multiple tap attributes on mobile devices.
  • TapNet's innovation lies in using inertial measurement unit (IMU) data (including accelerometer and gyroscope) as the primary input, supplemented by phone shape information. By sharing convolutional layers and output branches, TapNet achieves cross-device data joint learning and concurrent multi-task prediction.

• Implementation Steps and Key Techniques:

  • Signal Processing:
    • A simple gating mechanism filters out non-tap actions based on the z-axis signal of the accelerometer.
    • Extracts a 120-millisecond feature window and aligns it along the time axis.
  • Network Design:
    • A single-channel CNN efficiently extracts shape features from six-channel IMU signals, enabling multi-task learning through shared convolutional layers and branches.
    • Tasks include tap event classification (tap or not), tap direction classification, finger part classification, tap location classification, and tap location regression.
  • Model Training:
    • Training was conducted on a diverse single-user dataset (135,260 samples), ensuring data diversity through specific strategies.

Research Outcomes

• Specific Results:

  • Proposed a multi-task learning model, TapNet, capable of simultaneously recognizing multiple tap attributes and achieving cross-device joint learning.
  • Developed two datasets (a single-user dataset for training and a multi-user dataset for testing) and established benchmarks for off-screen tap recognition through experiments.

• Comparison with Existing Solutions and Advantages:

  • Compared to traditional machine learning or shallow neural networks in existing studies, TapNet improved accuracy by 51% in tap direction classification, 161% in tap location classification, and 30% in tap location regression.
  • TapNet significantly reduced runtime memory requirements and computational costs through shared computation compared to single-task models.

• Experimental or Evaluation Results:

  • TapNet demonstrated excellent performance on the multi-user dataset, particularly in high-complexity tasks such as tap direction and location classification.
  • Validated that models trained on single-user datasets achieved high cross-user generalization ability in simpler tasks like event detection and finger part classification.

• Limitations and Future Directions:

  • The current model may require further training to adapt to unseen devices, as cross-device adaptability is not yet fully realized.
  • High-resolution tap location classification requires collecting more refined multi-user data to enhance performance.
  • Plans include expanding the dataset and device range and developing open-source code to promote cross-domain applications and research.

Application Scenarios

• AssistiveTap: Provides single-handed interaction solutions based on back-tap and tilt operations.
• ExplorativeTap: Supports visually impaired users in selecting screen objects through combined screen and back-tap interactions.
• Interactive Wallpaper: Enables direct interaction with wallpaper objects, such as switching information on the lock screen.
• Inertial Touch: Assists in recognizing tap information when the touchscreen is non-functional or weakened (e.g., when wearing gloves).

Conclusion

TapNet explores the potential of off-screen interaction using smartphone IMU signals through deep learning, offering new possibilities for human-computer interaction design. The model successfully validates the potential of multi-task learning and cross-device training, expanding the design space for mobile interaction. Furthermore, this study provides reproducible benchmark results, datasets, and code foundations, advancing technological development in IMU sensor applications.