CamTroller: An Auxiliary Tool for Controlling Your Avatar in PC Games Using Natural Motion Mapping

Title of the Paper

CamTroller: An Auxiliary Tool for Controlling Your Avatar in PC Games Using Natural Motion Mapping

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Natural User Interface (NUI), Gesture Recognition, and Game Control
• Keywords: Natural Mapping, NUI, Motion Tracking, Intuitive Interaction, PC Games, RGB Camera, Keyboard-Mouse Assistance, Motion Controller, User Intuitiveness, Gaming Experience

Research Background and Issues

• Identified Problems or Challenges:

  1. The cognitive load, finger shortage, and high learning curve associated with keyboard and mouse operations in complex games.
  2. While motion control (e.g., motion-sensing devices) offers high "intuitiveness," it is difficult to adapt to current commercial PC games due to low compatibility with player needs and hardware.
  3. Prolonged keyboard and mouse usage may lead to health issues such as muscle fatigue and neck-shoulder injuries.

• Significance of the Problem: The limitations of traditional input devices and the increasing complexity of interaction demands can reduce gaming experiences and hinder players' performance in high-intensity or complex scenarios.

• Research Motivation and Related Work:

  • To explore auxiliary tools that leverage Natural User Interfaces (NUI) combined with natural mapping to resolve conflicts between complex game actions and traditional input systems.
  • Existing research often focuses on external motion devices, which are limited to specific platforms or entail high hardware costs, making them less widely applicable. This study aims to address these limitations using low-cost RGB cameras and natural motion mapping technology.

Solution

• Proposed Method or Solution:

  • A tool named "CamTroller" is proposed, which uses natural mapping technology to drive in-game virtual character actions through players' body postures (head and hand movements), while retaining keyboard and mouse as the primary input and CamTroller as a supplementary input.
  • MediaPipe, an open-source toolkit, is used to recognize players' head and hand postures via an RGB camera.
  • The game PUBG was chosen for experiments to validate the feasibility and user intuitiveness of the natural mapping interaction concept in real gaming scenarios.

• Innovative Contributions:

  1. Pioneering the use of built-in RGB cameras for posture tracking, achieving low-cost and highly compatible motion control.
  2. Addressing the frequent "finger shortage" issue in games by providing "incremental degrees of freedom" as supplementary input alongside keyboards and mice.
  3. Combining natural mapping theory to enhance the experience of one-to-one character control in traditional PC games.

• Implementation Steps and Key Technologies:

  1. Action Selection and Mapping:
     • Identify the actions CamTroller needs to process (e.g., leaning, crouching, jumping, free look, item usage).
     • Analyze detectable features of virtual character actions (e.g., head angles, hand positions, and gestures).
     • Match players' physically executable actions in the real world to the best corresponding virtual actions (achieving natural mapping).
  2. Technical Implementation:
     • Use the MediaPipe module to identify head and hand key points, estimating head positions and postures (including amplitude and speed).
     • Distinguish different item usage gestures through gesture recognition (e.g., bringing the hand close to the mouth to simulate drinking an energy drink, or to the chest to simulate adrenaline injection).
     • Optimize system sensitivity by setting thresholds, dead zones, and speed judgments to prevent accidental triggers and response delays.
     • Simulate keypress and mouse input signals to trigger in-game actions.
  3. Hardware and Validation:
     • Develop CamTroller using a standard RGB camera and commercial PC configurations, and validate its functionality and performance in PUBG.

Research Outcomes

• Specific Results:

  • Technical Feasibility Testing showed that CamTroller achieved an overall action recognition success rate of 99.33% across 11 designed actions, accurately mapping player movements to in-game characters.
  • Resource Efficiency: CamTroller's CPU usage during operation was low (5-10%), significantly lower than that of devices like Kinect (30-40%), making it more PC-friendly, with faster response times (approximately 0.033s to 0.04s).
  • User Study Results:
    • In objective performance tests, CamTroller's performance in "enemy observation and leaning movement" scenarios was comparable to professional players and outperformed average players.
    • In subjective user experience (evaluated using the QUESI questionnaire for interaction intuitiveness), CamTroller scored significantly higher than traditional keypress methods and professional player techniques.

• Advantages Over Existing Solutions:

  1. Compared to commercial devices like Kinect, it has a lower hardware barrier and requires no additional equipment.
  2. Compared to professional player techniques, it significantly improves ease of learning, helping beginners adapt quickly to competitive gaming scenarios.
  3. Effectively alleviates issues associated with keyboard and mouse input, such as "cognitive load," "finger shortage," and "repetitive strain injuries."

• Experimental or Evaluation Results:

  • In experiments involving item usage and leaning movements, CamTroller outperformed other methods across five dimensions of intuitiveness (subjective workload, goal achievement, learning effort, familiarity, and error perception).
  • Action trigger times (typically <0.7 seconds) met the requirements for smooth decision-making in games.

• Limitations and Future Directions:

  1. The system cannot perceive game states in real time, which may lead to overlooked virtual inputs.
  2. Experimental scenarios were relatively simple, and the performance advantages in complex gaming contexts remain unverified.
  3. Further analysis is needed for user learning curves, contextual adaptability, and personalized adjustments.
  4. Currently, CamTroller is designed for single-player, single-character games. Future work could extend its application to multi-character games or non-gaming scenarios (e.g., gesture-controlled video playback, document navigation).