"I Know What You Mean": Context-Aware Recognition to Enhance Speech-Based Games

Title of the Paper

“I Know What You Mean”: Context-Aware Recognition to Enhance Speech-Based Games

Paper Information

• Field of Study: Application of speech recognition technology in video games, with a focus on enhancing context awareness
• Keywords: Game design, speech recognition, speech-driven systems, voice-controlled games, speech interaction

Research Background and Problem

• Identified Issues:
  • Despite significant advancements in speech recognition technology in recent years, recognition errors persist, negatively impacting user experience. In games, such errors can increase player frustration, disrupt gameplay, and even lead to abandonment.
  • Voice-controlled games are rarely implemented and are often treated as optional features rather than core game mechanics.
• Significance:
  • Voice-controlled games offer new possibilities for intuitive and natural human-computer interaction. However, frequent recognition failures can severely undermine immersion and user satisfaction.
• Research Motivation:
  • To improve the performance of speech recognition systems by leveraging "context-aware" methods that integrate game environment and player action data, thereby enhancing the usability and player experience of speech-driven games.

Solution

• Methods and Approach:
  • A novel method called "context-aware speech recognition" is proposed, which improves command prediction accuracy by integrating the following three filters:
    • Scope Filter (measures the match between voice input and predefined commands based on Levenshtein distance).
    • Environment Filter (scores commands based on interactive objects visible from the player's perspective in the game).
    • Actions Filter (evaluates possible player actions on objects in the current state).
  • The most probable command is executed based on the weighted scores of these three filters.
• Innovations:
  • Introduction of environment and action filters to combine game states with potential player intentions, significantly reducing speech recognition errors.
  • First-time exploration of the practical effects of "context-aware speech recognition" in speech-driven games.
• Implementation Steps:
  1. Develop a 3D escape game called “Escape the Echo,” where players guide a virtual character, "Sophie," to complete tasks and achieve objectives.
  2. Integrate the "context-aware" module into the VOSK offline speech recognition engine for command processing.
  3. Design a controlled experiment with two groups: a control group using only Levenshtein distance for similarity calculation, and an intervention group using all three filters for command prediction.
  4. Evaluate player experience using standardized questionnaires (SUS and PENS) and custom questions, supplemented by semi-structured interviews for qualitative feedback.

Research Findings

• Key Results:
  • The context-aware method significantly reduced players' perceived recognition error rate. Players in the intervention group reported significantly higher overall game experience scores (from 5.35 to 6.30) and enjoyment scores (from 4.85 to 5.75) in the questionnaires.
  • The System Usability Scale (SUS) showed that the intervention group (85.12) perceived the system's usability to be significantly better than the control group (73.2).
  • Participants expressed higher satisfaction with the voice system, and the intervention group reported greater flexibility in voice operations compared to the control group.
• Advantages Compared:
  • Compared to traditional methods relying solely on text matching, the context-aware system demonstrated superior performance in terms of immersion and intuitive control.
  • Although the direct recognition rate was higher in the control group, the intervention group more effectively reduced misrecognitions and improved players' positive perceptions of the voice system.
• Experimental and Evaluation Results:
  • Game logs showed that context filters directly influenced 37% of predicted commands.
  • There were no significant differences in game completion time, total commands issued, or player performance, indicating that the method primarily improved user experience rather than performance.
• Limitations and Future Directions:
  • Limitations:
    • Limited sample size and the predominance of non-native English speakers among participants reduce the generalizability of the data.
    • The current command set and filter configurations are based on experimental formulas and lack automated optimization mechanisms.
    • The absence of animations during action execution affected the immersion for some players.
  • Future Directions:
    • Extend the research to more game types (e.g., 2D, fast-paced games) and device platforms (e.g., VR, mobile).
    • Explore multimodal inputs (e.g., gestures, gaze direction) to further support speech recognition.
    • Develop dynamic optimization models using deep learning to improve the accuracy of intent prediction by the filters.

Conclusion

This study provides an effective improvement to the recognition experience in speech-driven games through a context-aware method. It validates the feasibility of enhancing speech command system performance by integrating game environment and action data, thereby enriching research perspectives in the fields of speech interaction and game design.