Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on Touchscreens

Title of the Paper

Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on Touchscreens

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Optimization of User Interaction on Touchscreen Devices
• Keywords: Touch interaction, target selection, decision theory, reinforcement learning, suggestion mechanism, interaction techniques, deep Q-network, simulation experiments, user studies, intelligent interfaces

Research Background and Problem

• Problem or Challenge:

  • Accurate target selection on small touchscreen devices is challenging due to small target sizes and imprecise touch input, leading to frequent user errors.
  • Errors in selection result in inconvenient repeated operations and, in some cases, irreversible consequences, such as sending incorrect instant messages.
  • Suggestion mechanisms (displaying candidate target suggestions) can improve selection accuracy but may trigger too frequently, distract users, or reduce efficiency.
  • The key issue is determining when to provide suggestions and which candidate targets to include in the suggestions.

• Research Significance:

  • Enhancing the user experience on touchscreen devices.
  • Reducing the frequency of user errors and optimizing time costs.

• Motivation and Related Work:

  • Previous attempts have used probabilistic models, machine learning, and Bayesian inference to improve the accuracy of touchscreen target selection.
  • Methods such as magnification techniques (e.g., Shift) and target distribution-based suggestions have been applied for precise selection but still have room for improvement.
  • Reinforcement learning is increasingly being used to model user behavior and generate intelligent interactive interfaces.

Solution

• Method or Solution:

  • A reinforcement learning-based suggestion-based accurate target selection method (SATS) is proposed.
  • The target selection task is framed as a state-based sequential decision-making problem, where a computational agent dynamically decides whether to directly select a target or provide candidate suggestions based on input states.

• Innovation:

  • The problem is transformed into a Markov Decision Process (MDP), and reinforcement learning is used to generate optimal strategies, automatically optimizing when to provide suggestions and which targets to include.
  • Deep Q-Network (DQN) is employed as the foundational algorithm, training the model through simulated interactions between users and the environment.
  • This is the first study to apply reinforcement learning to suggestion-based target selection research.

• Implementation Steps and Key Techniques:

  1. Environment Modeling:
     • Touch point distribution simulation: A dual Gaussian model is used to represent user touch behavior.
     • Time cost estimation: An empirical time model for target selection is constructed based on experimental data.
  2. Reinforcement Learning Framework:
     • Define key components such as problem states, actions, and rewards:
       • State: Target layout and user touch points.
       • Action: Directly select a target or provide suggestions for a varying number of targets.
       • Reward: Negative reward design based on the time cost of selection operations.
     • Use a deep Q-network to learn the agent's optimal strategy.
  3. User Testing and Experimental Evaluation:
     • Provide simulated environments with various target layouts and sizes.
     • Compare error rates, selection times, and subjective evaluations across different methods.

Research Outcomes

• Specific Results:

  • SATS demonstrated significantly reduced error rates and shorter selection times in experiments:
    • Compared to direct selection methods (e.g., BayesianCommand), SATS achieved a substantial reduction in error rates with a slight increase in time cost.
    • Compared to magnification methods (e.g., Shift), SATS reduced both error rates and selection times.
    • Outperformed existing suggestion mechanisms (e.g., MUCS) in balancing selection accuracy and time cost more efficiently.
  • The learning strategy of SATS dynamically adapts to different target distributions in various scenarios, leading to more optimal operations.

• Advantages:

  • Dynamic decision-making for triggering suggestions significantly improved selection accuracy.
  • User preference surveys indicated that SATS outperformed existing methods in subjective evaluations.

• Experiments or Evaluations:

  • User studies (18 participants) showed that SATS effectively reduced error rates (from 3.64% to 0.89%) compared to other methods.
  • The suggestion triggering conditions of SATS closely matched real user needs, achieving high coverage while reducing input ambiguity.

• Limitations and Future Directions:

  • Limitations: The current strategy is derived from a pre-training process and does not dynamically adapt based on individual user data.
  • Future Work:
    • Explore strategy fine-tuning based on personalized user data.
    • Extend SATS technology to non-touchscreen scenarios (e.g., gaze-based or gesture-based interactions).
    • Further optimize real-time prediction performance on hardware devices.

This structured analysis highlights the core themes, content, and contributions of the paper, enabling researchers to quickly grasp the key points and support further exploration and application.