LLM Powered Text Entry Decoding and Flexible Typing on Smartphones

Research Background and Problem

• Problem and Challenges: This paper highlights that while large language models (LLMs) perform exceptionally well in various language tasks, their application in keyboard decoding—converting touch input signals such as taps and gestures into text—remains underexplored. Current keyboard decoding methods typically support only a single input type (e.g., taps or gestures alone) and are unable to handle complex hybrid inputs (a combination of taps and gestures).
• Importance: Supporting hybrid input types is crucial for improving text input adaptability and user experience on mobile devices, especially in scenarios involving long words or complex phrase inputs.
• Research Motivation and Related Work:
  • This study is inspired by the limitations of existing tap and gesture decoders (e.g., SHARK² and statistical decoders) and aims to develop a novel decoder capable of supporting multiple input modes simultaneously.
  • Previous work has primarily focused on single input modes (e.g., pure gestures or pure taps) without addressing the complexity of hybrid inputs.

Solution

• Method and Approach:

  • A keyboard decoder based on an LLM (FLAN-T5 small model) is proposed.
  • A "Flexible Typing" method is introduced, allowing users to input text flexibly using pure taps, pure gestures, multi-segment gestures, or hybrid inputs combining taps and gestures.

• Innovations:

  • Support for Diverse Inputs: A new decoder integrating multiple input methods is proposed, supporting taps, single gestures, multi-segment gestures, and their combinations.
  • Robust Decoding: Leveraging the advanced architecture of the FLAN-T5 model, the decoder achieves high accuracy and robustness when processing complex input signals such as hybrid inputs.
  • Infrastructure Improvements: The model is fine-tuned using large-scale real and synthetic data, enabling it to handle both traditional tasks and uncertain, complex hybrid inputs.

• Implementation Steps:

  1. Model Selection and Design:
     • A pre-trained FLAN-T5 small language model is used as the foundation due to its sequence-to-sequence (Seq2Seq) architecture, which is suitable for keyboard decoding tasks.
  2. Input Processing:
     • User input trajectories (including tap points and gesture paths) are converted into 32 equidistant points and mapped to the nearest letter sequence on the keyboard.
     • Contextual prefixes are added to each input sequence to enhance the model's understanding of the task.
  3. Data Collection and Processing:
     • Tap and gesture inputs are extracted from existing real-world datasets.
     • A batch of new flexible input data is synthesized, covering multi-segment gestures and tap-gesture hybrid types.
  4. Model Fine-Tuning and Training:
     • The FLAN-T5 model is fine-tuned on large-scale hybrid input data, with efficient training conducted on a single GPU.
  5. Decoding Process:
     • An extended Beam Search algorithm is used to generate the top four candidate words.
     • During actual operation, the top candidate word is directly displayed in the keyboard input bar, while the remaining candidates are provided as suggestions to the user.

Research Outcomes

• Specific Results:

  • The model performs exceptionally well across various input types:
    • Gesture Input: Achieved a Top-1 accuracy of 93.1%, nearly 20% higher than the general SHARK² decoder.
    • Tap Input: On real tap datasets, the model performed comparably to statistical decoders, with a Top-1 accuracy of 95.4%.
    • Hybrid Input: Achieved a Top-1 accuracy of 88.0% on simulated datasets, a 21.2% improvement over the SHARK² extended decoder's 72.6%.
  • In user studies, flexible input received positive feedback, with user preference ratings reaching a perfect score of 5 out of 5.

• Advantages:

  • Significantly improved decoding accuracy, particularly in scenarios involving complex input signals such as hybrid tap-gesture inputs.
  • Greater flexibility to accommodate diverse user input needs, such as long words and faster input scenarios.

• Experimental Evaluation Results:

  • In real user scenarios, the model's decoding error rate (Top-1 WER) was significantly lower than the traditional SHARK² model (5.9% vs. 20.7%), with further improvements observed in the Top-4 candidate words.
  • The flexible input method received strong support in user studies, with 35.9% of words entered via gestures, 29% via taps, and 29% using a combination of both.

• Limitations and Future Directions:

  • Limitations:
    • The model still relies on a static dictionary, which may result in poor performance for out-of-vocabulary words (e.g., passwords, proper nouns, new terms).
    • The current implementation requires users to manually press the spacebar to confirm word completion, potentially affecting input speed.
  • Future Improvements:
    • Develop dynamic dictionary extensions to support adaptive personalized vocabulary.
    • Explore character-level or gesture-level editing methods to enhance fine-grained editing capabilities.
    • Optimize the integration of cloud-based and local deployment strategies while evaluating the potential of more powerful LLMs (e.g., GPT-4 or Llama2).

Conclusion

This study introduces an LLM-based decoder that significantly enhances the performance of traditional keyboard decoders and enriches the text input experience on smartphones through the flexible input method. It provides valuable insights and directions for the future design of keyboard input systems.