SilentSpeller: Towards mobile, hands-free, silent speech text entry using electropalatography

Title of the Paper

SilentSpeller: Towards mobile, hands-free, silent speech text entry using electropalatography

Bibliographic Information

• Research Area: Human-Computer Interaction and Text Input Technology
• Keywords: Wearable Computing, Silent Speech Interface, Text Input, Tactile Sensing, Human-Computer Interaction, Mobile Technology

Research Background and Issues

• Identified Problems and Challenges:

  • Voice input is unsuitable in many situations, such as when privacy is required or under social constraints.
  • Existing silent speech recognition technologies are limited by vocabulary size (typically supporting only around 100 words) and require users to remain stationary.
  • Few studies focus on real-time input testing, as most methods are confined to offline experiments, potentially overestimating their effectiveness.

• Significance of the Research:

  • Provides an alternative input method for users with limited finger dexterity (e.g., those with muscular dystrophy, multiple sclerosis, etc.).
  • Addresses the limitations of manual input and the challenges of using voice control in social settings, enhancing input efficiency and privacy protection.

• Motivation and Related Work:

  • While voice input offers advantages in convenience and learnability, existing silent speech recognition methods rely on specific devices, require extensive training data, and lack user independence across different individuals.

Solution

• Method and Approach:

  • Developed the SilentSpeller system, enabling users to input text by silently spelling letters.
  • Utilized a device called SmartPalate, consisting of a dental retainer embedded with 124 capacitive touch sensors to monitor tongue movements against the hard palate.

• Innovations:

  • Introduced silent spelling as an alternative to silent speech, improving recognition rates and vocabulary generality through letter-by-letter input.
  • Supported a dictionary of over 1,000 words and enabled usage while walking.
  • Provided a real-time input experience with stable operation in mobile and dynamic scenarios.

• Implementation Steps and Key Technologies:

  1. Hardware Utilization:
     • The SmartPalate device samples tongue activity in real-time at a frequency of 100 Hz.
  2. Data Compression:
     • Applied Principal Component Analysis (PCA) to reduce 124 original sensor signals to 16 components ("eigen-palates").
  3. Recognition Model:
     • Used Hidden Markov Models (HMM) for temporal pattern matching to decode spelling into text.
  4. User Training:
     • Trained user recognizers with 2,328 words and an additional 107 short phrases.
  5. Dictionary Expansion:
     • Supported generalization tests for unseen words and allowed input of user-specific proper nouns.

Research Results

• Specific Outcomes:

  • In offline word input experiments, the average character accuracy for a 1,164-word vocabulary reached 97%.
  • Training data traversal showed optimal learning effects with approximately 1,500 training words.
  • When testing user input for unknown words, the system achieved an average character recognition rate of 94.5%, demonstrating a certain degree of generalization capability.
  • Input accuracy was almost unaffected by walking versus sitting scenarios (96.5% sitting, 97.5% walking).
  • Real-time input experiments achieved 37 words per minute with an accuracy of 87%.

• Advantages Compared to Existing Solutions:

  • More suitable for operation during movement compared to other silent speech interfaces.
  • Lower user training time cost, making it ideal for personalized input systems.
  • Offers real-time input functionality, enhancing practical applicability compared to offline tests.

• Experimental Results:

  • Performance in multi-user comparisons was close to the average input speed of smartphone virtual keyboards (embedded QWERTY at 48 wpm).
  • Users showed significant improvement in familiarity with SilentSpeller over the course of the experiments, with continuous performance enhancement trends.

• Limitations and Future Directions:

  • Currently requires users to wear specialized hardware (dental retainer), limiting its adoption in social environments.
  • Lacks user-independent training models; future work will explore user adaptability improvements based on big data.
  • Hardware improvements could include adding lip motion capture sensors, adopting wireless communication modules (e.g., Bluetooth Low Energy), and optimizing device comfort for wearability.