Communication, Collaboration, and Coordination in a Co-located Shared Augmented Reality Game: Perspectives From Deaf and Hard of Hearing People

Document Title

Communication, Collaboration, and Coordination in a Co-located Shared Augmented Reality Game: Perspectives From Deaf and Hard of Hearing People

Document Information

• Topic Area: Augmented Reality (AR), Accessibility Design, Technology Experiences of Deaf and Hard of Hearing People
• Keywords: Deaf and Hard of Hearing, Collaborative Augmented Reality, Shared Augmented Reality, Multiplayer Games, Accessibility Design, User Experience, Multimodal Communication, Collaboration, Mixed Reality, Game Design

Research Background and Issues

• Problems or Challenges Identified by the Authors:

  • Existing research on Shared Augmented Reality (S-AR) and Collaborative Augmented Reality (C-AR) predominantly focuses on design, technical implementation, and user interfaces, with insufficient attention to accessibility and inclusivity for Deaf and Hard of Hearing (DHH) groups.
  • There is a scarcity of studies in the current literature examining the communication, collaboration, and coordination behaviors of DHH groups in co-located collaborative augmented reality environments.
  • AR environments that primarily rely on auditory and visual modalities may exacerbate social isolation among DHH individuals, impacting equitable participation in education, entertainment, and other domains.

• Importance of the Research:

  • Understanding and improving the experiences of DHH individuals in collaborative augmented reality environments can help create more inclusive technological applications, with broad implications for game development, educational training, industrial applications, and beyond.

• Research Motivation:

  • To enhance accessibility for DHH groups, the study aims to integrate real-world communication strategies—often reliant on visual cues and gestures—with AR technologies, exploring the applicability and limitations of collaborative augmented reality in multi-user scenarios.

Solution

• Proposed Method or Solution:

  • The authors designed and tested a multiplayer collaborative co-located augmented reality game, "Urban Legends," and conducted user experiments and semi-structured interviews with 17 DHH participants to analyze their communication, collaboration, and coordination behaviors in the collaborative augmented reality environment.

• Innovative Aspects:

  • Investigated how DHH individuals collaborate in co-located augmented reality environments without embedded communication technologies (e.g., built-in chat or voice communication), utilizing multimodal communication methods such as speech, gestures, and body language.
  • Summarized key challenges and design recommendations for DHH users, including features like automatic speech-to-text generation and customizable visual cue information.

• Implementation Steps and Key Technologies:

  • Conducted five rounds of game experiments using the "Urban Legends" game, each lasting approximately 240 seconds, divided into public and private spaces.
  • Analyzed communication and collaboration strategies before and after gameplay, employing Retrospective Thematic Analysis (RTA) to code and extract themes from interview data.
  • Recorded participant behaviors both within the game (interactions with virtual objects and environments) and outside the game (collaboration with peers and the surrounding environment).
  • Data collection tools included video observation, device screen recordings, and interview transcripts.

Research Findings

• Key Findings:

  • Communication Behavior:

    • DHH users effectively utilized multimodal communication methods, including speech, gestures, body language, and speech-to-text applications on mobile devices.
    • Communication frequency decreased during gameplay, with most users relying on body language and gestures for simple commands, while more complex strategy discussions were concentrated before each game round.

  • Collaboration Behavior:

    • Collaboration was more prominent during pre-game discussions, where users assigned roles and formulated strategies.
    • Collaboration decreased as game rounds progressed, due to increased familiarity with the rules and limitations imposed by the game mechanics on collaborative requirements.

  • Coordination Behavior:

    • Coordination with AR components: Users needed to continually familiarize themselves with game rules and perform physical movements to achieve objectives.
    • Coordination with the environment: Open and spacious settings helped reduce distractions and improve safety, though public areas introduced potential interruptions from passersby.
    • Coordination with peers: Communication and collaboration improved gradually, even among groups of strangers meeting for the first time.

  • Design Recommendations:

    • Enhance communication support, such as integrating in-game speech-to-text, gesture menus, and customizable visual information.
    • Provide clearer visual feedback, including visual cues for enemy locations and game objectives.
    • Offer safer and mobility-friendly physical spaces, while designing game tasks that can be completed with minimal communication.

• Advantages Over Existing Solutions:

  • Addressed the gap in augmented reality research regarding DHH users, particularly capturing detailed communication behaviors in dynamic co-located collaborative gaming scenarios.

• Experimental or Evaluation Results:

  • User experiments and interviews revealed insights into multimodal communication, spatial preferences, and critical design implications for game accessibility.
  • Accessible game environments facilitated smoother communication and personalized coordination experiences.

• Limitations and Future Directions:

  • The study participants were limited to university students, many of whom had prior knowledge of AR technologies, potentially restricting the generalizability of the findings.
  • The research did not include hearing participants for comparative analysis, focusing solely on the unique needs of DHH groups.
  • Future work will explore collaborative augmented reality applications across different domains and expand the participant pool to include diverse ages, professions, and technical backgrounds, further validating the study's findings.