KeySense: LLM-Powered Hands-Down, Ten-Finger Typing on Commodity Touchscreens

Paper Title

KeySense: LLM-Powered Hands-Down, Ten-Finger Typing on Commodity Touchscreens

Publication Info

• Topic area: Enabling ergonomic and efficient ten-finger typing on touchscreens using software-only solutions.
• Keywords: Touchscreen typing, ten-finger typing, LLM decoding, ergonomic text entry, touch event clustering, synthetic training data, cognitive window, FLAN-T5-small, human-computer interaction, text entry systems.

Background and Problem

• Problem / challenge: Current touchscreen keyboards require users to hover their fingers, leading to fatigue and preventing the use of familiar ten-finger typing motor skills. Touchscreens lack the ability to distinguish intentional taps from resting-finger noise, making hands-down typing ambiguous.
• Significance: Addressing this issue could enable faster, more comfortable, and ergonomic typing on commodity touchscreens, leveraging users' existing muscle memory from physical keyboards.
• Motivation and related work: Previous solutions relied on specialized hardware (e.g., pressure-sensitive surfaces, depth cameras) or simplified software heuristics, which are impractical for commodity devices. Statistical and probabilistic decoders have been explored but fail to fully resolve the ambiguity caused by resting fingers. This paper aims to achieve hands-down typing using only software and commodity hardware.

Solution

• Proposed approach: KeySense, a software-only system that combines heuristic filtering of touch events with a fine-tuned LLM (FLAN–T5-small) to decode noisy letter sequences into intended words.
• Novelty:
  1. A lightweight pre-decoder that clusters touch events within a 100 ms cognitive window and selects intentional taps using a reach-aware heuristic.
  2. A fine-tuned FLAN–T5-small LLM trained on synthetic noisy-clean word pairs to correct letter sequences.
  3. Demonstration of hands-down, ten-finger typing on commodity, pressure-insensitive touchscreens without additional hardware.
  4. First detailed characterization of hands-down typing behavior on passive surfaces.
• Procedure and key techniques:
  1. Touch events are aggregated into threads and grouped into clusters based on a 100 ms cognitive window.
  2. A travel score heuristic selects the most likely intentional touch within each cluster.
  3. A synthetic corpus of noisy-clean word pairs is generated to train the FLAN–T5-small LLM for sequence correction.
  4. The system processes touch input into a provisional letter sequence, which is decoded into the final word by the LLM.

Results

• Concrete findings:
  • The fine-tuned FLAN–T5-small achieved 84.8% Top-1 accuracy, outperforming statistical baselines (75.7% and 79.3%).
  • In a user study, participants typed faster with KeySense (28.3 WPM vs. 26.2 WPM, p < 0.01) and rated it as less physically demanding (NASA-TLX median 1.5 vs. 4.0).
  • Decoding latency was low, with an end-to-end delay of 110 ms, well below perceptual thresholds.
• Advantage over baselines:
  • KeySense's LLM decoder outperformed Bayesian and n-gram decoders, particularly on medium and long words.
  • The system enabled hands-down typing without requiring specialized hardware, unlike prior systems.
• Experiments / evaluation:
  • A 12-participant study compared KeySense to conventional hover typing on transcription tasks.
  • Metrics included typing speed (WPM), word error rate (WER), corrected error rate (CER), and subjective usability ratings.
  • Synthetic data and real-world touch logs were used to train and evaluate the system.
• Limitations and future work:
  • Current implementation focuses on word-level correction and English text, limiting multi-word context and handling of out-of-vocabulary (OOV) words.
  • Future work includes extending to sentence-level decoding, hybrid decoders, personalization of timing parameters, and longitudinal studies.

Summary

KeySense introduces a software-only solution for hands-down, ten-finger typing on commodity touchscreens by combining heuristic touch filtering and a fine-tuned FLAN–T5-small LLM. It achieves high decoding accuracy (84.8% Top-1) and ergonomic benefits, enabling faster and less physically demanding typing compared to conventional methods. The system demonstrates the feasibility of restoring physical keyboard motor skills on touchscreens without additional hardware. Future work aims to enhance context-aware decoding, handle OOV words, and explore hybrid approaches for broader applicability.