Juggling Extra Limbs: Identifying Control Strategies for Supernumerary Multi-Arms in Virtual Reality

Research Background and Issues

• Identified Problems or Challenges:

  • Utilizing supernumerary limbs (SLs) to accomplish complex tasks is a prominent topic in virtual reality (VR) and robotics research. However, most current control strategies rely on fixed mapping methods, which struggle to adapt to dynamic environments and complex tasks.
  • There is a lack of research on how users coordinate their own limbs with semi-autonomous virtual supernumerary limbs (VSLs) to complete tasks, especially as task complexity and autonomy levels increase.

• Importance of the Issue:

  • Investigating interaction strategies for supernumerary limbs contributes to the development of more efficient human-computer interaction systems, enhancing human capabilities to tackle more complex tasks.
  • Exploring control strategies for virtual limbs in VR environments is not only significant for future VR applications but also provides valuable insights for the design and development of physical robotic limbs.

• Research Motivation and Related Work:

  • Current research on supernumerary limbs predominantly relies on fixed mapping methods, which lack flexibility and fail to support dynamic task switching.
  • Considering the potential of semi-autonomous systems, this study aims to explore how users adopt diverse interaction strategies in virtual environments with varying levels of automation.
  • Previous studies have shown that VR-based experiments can serve as an important tool for understanding users' body self-perception and behavior in capability enhancement scenarios.

Solution

• Methods or Solutions:

  • Employing the Wizard-of-Oz method to simulate semi-autonomous VSLs, collecting user data during basic control tasks and factory tasks to analyze interaction strategies.
  • Investigating two control conditions: low autonomy (requiring detailed interaction instructions) and high autonomy (supporting complex, multi-step tasks).

• Innovations:

  • Proposed four interaction guidance strategies: instruction, demonstration, task delegation, and object annotation, which enable flexible adaptation to task complexity or system requirements.
  • Using a multi-layered approach (e.g., qualitative interviews and quantitative evaluations) to reveal how automation can be adjusted to meet user needs, thereby improving the design efficiency of multi-limb systems.

• Implementation Steps and Key Technologies:

  1. Experiment Design: Develop a VR environment based on Unity, supporting synchronized interaction between two Oculus Quest devices to simulate user and operator interactions; partitioning to conceal the presence of the human operator.
  2. Task Setup: Include basic control tasks (e.g., button switching) and "factory tasks," requiring users to simultaneously use their controlled virtual limbs and limbs controlled by VSLs.
  3. Data Collection: Gather user experience and behavioral data through recordings, questionnaires, and task performance metrics (completion time, error rate, etc.).
  4. Error Simulation: Introduce intentional operational errors to simulate real-world system autonomy deficiencies.

Research Outcomes

• Specific Findings:

  • Interaction Strategies:
    • Most users preferred direct control under low autonomy conditions, while task delegation was favored in high autonomy environments.
    • Annotation and action demonstration were considered efficient ways to manage tasks but required precise commands and adaptability.
  • User Experience:
    • In low autonomy environments, users exhibited stronger perceptions of body ownership and control over virtual limbs.
    • High autonomy conditions resulted in higher task efficiency but reduced users' sense of control and ownership over virtual limbs.
  • Performance Improvements:
    • High autonomy conditions significantly reduced task completion times and dramatically decreased error rates in complex scenarios.

• Advantages Over Existing Solutions:

  • Compared to fixed mapping schemes, the dynamic control approach in this study enabled users to better adapt to changing task demands.
  • High autonomy environments fully leveraged the efficiency of virtual multi-limb systems, enhancing performance in complex tasks.

• Experimental Results:

  • In basic tasks, high autonomy significantly shortened task completion times (p < 0.05).
  • In factory tasks, error rates were lower under high autonomy conditions (p < 0.01), demonstrating its advantages in executing complex tasks.
  • Results from the "Embodiment Questionnaire" indicated that users experienced stronger ownership and control of virtual limbs under low autonomy conditions.

• Limitations and Future Directions:

  • Limitations:
    • The autonomous system in the experiment was simulated using the Wizard-of-Oz method, which may differ from real autonomous robotic systems.
    • The sample size was small (14 participants) and primarily consisted of users with VR experience, which may limit the generalizability of the findings to users without such experience.
    • All research was conducted in VR environments, which, while informative, requires further validation for physical robotic applications.
  • Future Directions:
    • Investigate additional levels of automation to explore their impact on multi-scenario, diverse task execution.
    • Extend research to real-world applications of physical robotic limbs, particularly in industrial production and medical rehabilitation.
    • Examine the generalizability and effectiveness of findings across larger and culturally diverse user groups.