Exploring Deaf And Hard of Hearing Peoples’ Perspectives On Tasks In Augmented Reality: Interacting With 3D Objects And Instructional Comprehension

Research Background and Issues

What problems or challenges did the authors identify?

• Augmented Reality (AR) is typically designed to cater to users with uniform sensory abilities, potentially overlooking the unique needs of Deaf and Hard of Hearing (DHH) users.
• DHH users may face distinct challenges in AR environments, especially for tasks that heavily rely on sensory input (e.g., 3D object interaction and instruction comprehension).
• DHH individuals primarily rely on visual and tactile input, and this sensory limitation may lead to higher cognitive load and operational difficulties.

Why is this issue important?

• Over 5% of the global population (approximately 430 million people) belongs to the DHH community, who may face significant risks of a digital divide when accessing sensory-rich technologies like AR.
• Current research on AR interaction design for the DHH community is limited. Addressing these issues not only enhances technology accessibility but also creates more opportunities for learning, training, and employment.

Research Motivation and Related Work

• The authors note that existing research focuses on facilitating communication between DHH users and hearing users, as well as enhancing DHH users' situational awareness. However, studies on complex tasks such as placing and rotating 3D objects remain underexplored.
• Applying existing AR technologies to contexts like education, entertainment, and vocational training requires a deeper understanding of the needs and challenges of the DHH community.

Solutions

What methods or solutions did the authors propose?

• Using "Angry Birds AR" as a research tool to explore the experiences and challenges of DHH and hearing users in AR environments.
• Collecting feedback from DHH and hearing participants to identify interaction barriers and propose design recommendations to improve AR accessibility.
• Key design focuses include customizable user interfaces, modular feedback systems as alternatives to audio feedback, and virtual sign language avatars.

What is innovative about this solution?

• Conducting user experience research for DHH individuals in a gamified environment. Mini-games simplify tasks to test target users' operational behaviors in AR environments.
• Comparative analysis with hearing users to deepen understanding of the unique interaction needs of DHH users.
• Proposing specific design recommendations, such as providing sign language virtual assistants and customizable tactile feedback for DHH users.

What are the implementation steps? What key technologies were used?

1. Data Collection: The study involved gameplay sessions and short-term and long-term interviews with participants.

   • Participants played AR games in both private labs and public spaces, with their experiences and feedback recorded.
   • Reflexive Thematic Analysis (RTA) and triangulation methods were used to integrate data from various sources (game recordings, interview transcripts, and observation notes).

2. Participant Diversity: Data was collected from 11 DHH and 15 hearing participants. DHH participants were further categorized into groups preferring sign language or English.

3. Technical Analysis: Specific analysis of user interaction methods in the game (e.g., physical motion and button interactions), sensory feedback (tactile vibrations and audio cues), and comprehension of textual instructions.

Research Findings

What specific findings were achieved?

• Identified specific needs and barriers for DHH users in interaction, instruction comprehension, and feedback mechanisms:
  • Interaction Methods: DHH users preferred using UI elements for interaction in public spaces, while in private spaces, they were more willing to combine physical motion with interface buttons.
  • Sensory Feedback: The absence or improper design of tactile feedback reduced game immersion.
  • Instruction Presentation: While textual instructions were generally accepted, their short display time caused some users to miss critical information.
  • Balance of Visual and Audio Feedback: DHH users could not effectively utilize audio feedback, necessitating more prominent and persistent visual cues.

How does it compare to existing solutions?

• The proposed design recommendations emphasize modality switching (visual, tactile, auditory), providing diverse support for the sensory preferences of DHH users.
• The gamified environment (e.g., using Angry Birds AR) closely mirrors real-world user experiences, simplifying experimental complexity.
• Focused on contextual user experiences, such as the differences in spatial awareness needs in public environments versus immersion needs in private settings.

What were the experimental or evaluation results?

• Interaction Preferences: Most DHH users prioritized environmental awareness in public spaces, preferring to remain stationary, while hearing users prioritized immersive experiences, utilizing more physical movement.
• Feedback Mechanisms: Tactile feedback significantly enhanced immersion but was underutilized. Weak audio feedback did not noticeably improve the experience for hearing users.
• Instruction Display Issues: Current instruction display times were too short, causing some users to miss them; DHH users suggested persistent display options.

Limitations and Future Directions

• Limitations:

  • Participants were primarily university-affiliated individuals, resulting in limited sample diversity.
  • The game tasks were relatively simple, failing to test more complex AR application scenarios.
  • Some DHH participants were already familiar with traditional versions of Angry Birds, which may have influenced the study results.

• Future Directions:

  • Extend the research to more complex industrial applications or educational scenarios.
  • Recruit participants from a broader range of age groups and professional backgrounds to enhance the applicability of findings.
  • Investigate AR applications involving high cognitive load and multitasking.
  • Collaborate closely with the DHH community, adopting participatory design methods to more accurately reflect user needs.

Conclusion

This study explores the challenges faced by DHH users in AR environments and proposes targeted design improvements, advancing research on the accessibility of augmented reality technologies. This work not only provides valuable insights for the gaming field but also lays the foundation for applications in education, industrial training, and other practical domains.