CRTypist: Simulating Touchscreen Typing Behavior via Computational Rationality

Title of the Paper

CRTypist: Simulating Touchscreen Typing Behavior via Computational Rationality

Paper Information

• Research Domain: Human-Computer Interaction (HCI), simulation and modeling of touchscreen typing behavior
• Keywords: simulation model, reinforcement learning, touchscreen typing, human-computer interaction, cognitive modeling, individual differences, keyboard layout, auto-correction

Research Background and Problem Statement

• Problems and Challenges

  • Touchscreen typing is a complex task requiring coordination between visual attention and finger movements, involving rapid input, proofreading, and error correction in a cyclic control process.
  • The diversity of keyboard types and user behaviors makes it difficult for existing computational models to effectively predict typing performance and strategy changes.
  • Current models heavily rely on experimental data or specific tasks, lacking generalizability and struggling to predict behavior in new environments, such as changes in keyboard design, user capabilities, or the impact of assistive features.

• Significance of the Research

  • Simulating human touchscreen typing behavior can help improve keyboard layouts, enhance user experience and accessibility, and provide performance predictions during the design phase, reducing development costs.
  • A model with generalizability and the ability to predict individual differences can offer solutions for supporting special user groups (e.g., those with visual or memory impairments).

• Motivation and Related Work

  • This study is inspired by the limitations of existing models, such as classical models based on Fitts' law or cognitive architecture models (e.g., GOMS and EPIC), which require extensive manual design.
  • Recent studies have introduced preliminary supervisory control models, but they still face challenges in extending to new keyboard layouts, addressing individual differences, and generating realistic behaviors.

Solution

• Methods and Approach

  • A novel computational model, CRTypist (Computational Rational Typist), is proposed to simulate touchscreen typing behavior.
  • CRTypist operates directly on pixel data without requiring manual design of keyboard features, achieving flexibility through a modular hierarchical architecture.
  • It adopts the principle of computational rationality, viewing typing as an optimal behavior process constrained by vision, motor, and memory limitations.

• Innovations

  • The core innovation lies in redefining the supervisory control problem: continuously updating time-decayed beliefs about the entered text via a working memory module to guide visual and finger movement decisions.
  • A modular architecture is employed, including a visual module (simulating both central and peripheral vision), a finger module, and a working memory module for layered modeling.

• Implementation Steps and Techniques

  • Step-by-step training:
    1. Pre-train the visual, finger, and working memory modules to form an intermediate environment.
    2. Optimize strategies to adapt to diverse cognitive parameters and keyboard designs.
    3. Use Bayesian optimization to fit model parameters, ensuring generated behaviors align with real user data.
  • Train decision-making strategies using reinforcement learning algorithms (e.g., Proximal Policy Optimization, PPO).
  • Develop a real-world data benchmark (MobileTyping) for model evaluation.

Research Outcomes

• Specific Results

  • CRTypist achieves high accuracy in simulating typing behaviors, including single-finger typing and two-thumb typing, closely replicating human performance in text input speed, error rates, and proofreading strategies.
  • It can automatically adapt to various keyboard layouts (e.g., CHUBON and KALQ) and features (e.g., auto-correction), while also supporting predictions of individual differences.

• Advantages Compared to Existing Solutions

  • Compared to baseline models (e.g., Optimal Supervisory Control models), CRTypist demonstrates greater adaptability across diverse keyboard designs.
  • It does not rely on manually designed state-action features and operates directly on screen pixels, enhancing flexibility in user modeling.
  • A single model supports large-scale individual data, covering nearly 97% of single-finger and two-thumb typing speed distributions.

• Experimental and Evaluation Results

  • CRTypist performs exceptionally across multiple tasks in the MobileTyping benchmark, excelling in generating realistic behaviors, capturing individual differences, and generalizability.
  • Simulated user behaviors show consistent average error rates and speeds across various keyboards, aligning well with experimental data and demonstrating the model's reliability and generalization capabilities.
  • Predictions for auto-correction functionality indicate a typing speed increase of approximately 2 WPM, fitting well with real-world data.

• Limitations and Future Directions

  • Limitations: The current model can only predict post-practice performance and does not cover skill acquisition processes; it cannot simulate the needs of extreme user groups (e.g., visually impaired users).
  • Future directions:
    • Extend the model to include long-term memory, reading behavior, and task learning capabilities.
    • Optimize supervisory control assumptions and investigate the impact of additional device features on user behavior.
    • Develop more advanced HCI simulation tools for intelligent responses and multitasking scenarios.

Conclusion

Built on a modular, flexible architecture and pixel-based operation, CRTypist provides a groundbreaking solution for simulating touchscreen typing behavior. It offers comprehensive support for capturing individual differences, adapting to keyboard layouts, and generating realistic behaviors, presenting significant potential for design and research in the field of human-computer interaction.