AuslanSpell: An Interactive Technology for Improving Auslan Fingerspelling Comprehension

Paper Title

AuslanSpell: An Interactive Technology for Improving Auslan Fingerspelling Comprehension

Publication Info

• Topic area: Interactive technology for sign language education
• Keywords: Auslan, fingerspelling, sign language, motion capture, 3D animation, Deaf education, interactive learning, avatar technology, usability testing, language acquisition

Background and Problem

• Problem / challenge: Auslan learners face significant challenges in decoding fingerspelling due to limited practice opportunities and the lack of interactive, dynamic resources tailored to Auslan's two-handed alphabet.
• Significance: Addressing these challenges is crucial to improving Auslan education, increasing the number of certified interpreters, and reducing isolation for Deaf individuals in professional and social contexts.
• Motivation and related work: While digital tools for ASL fingerspelling exist, they are non-transferable to Auslan due to differences in alphabets. Current Auslan resources are static and lack engagement, failing to demonstrate natural transitions between letters. Prior studies highlight the importance of transitions and movement in fingerspelling comprehension, but no Auslan-specific interactive tools have been developed.

Solution

• Proposed approach: AuslanSpell, an interactive application that generates 3D animated Auslan fingerspelling sequences from arbitrary English text inputs, featuring smooth transitions and customizable settings.
• Novelty:
  1. Development of the first 3D Auslan fingerspelling dataset using motion capture technology.
  2. Creation of a freely accessible application capable of producing realistic Auslan fingerspelling animations.
  3. Validation of AuslanSpell as an effective learning tool for novice signers, demonstrating its potential to enhance comprehension and engagement.
• Procedure and key techniques:
  • Motion capture of 8 Deaf Auslan signers to record handshapes and transitions.
  • Iterative validation and cleaning of animations with Deaf educators to ensure accuracy.
  • Development of the application with features like adjustable signing speeds, rotatable views, and customizable hand models.
  • User evaluation with 33 novice signers to assess learning outcomes and usability.

Results

• Concrete findings:
  • Participants scored an average of 8.2/10 on multiple-choice readback tasks, significantly above chance.
  • On free-text tasks, participants identified first letters correctly 80% of the time, last letters 74%, and middle letters 34%.
  • Mean System Usability Scale (SUS) score: 81.2/100, indicating high satisfaction.
• Advantage over baselines:
  • Improved engagement and comprehension compared to static resources, with transferable skills from the application to human fingerspelling readback.
  • Features like adjustable speeds and rotatable views were highly rated by users.
• Experiments / evaluation:
  • Participants: 33 novice signers (university students with no prior Auslan experience).
  • Tasks: Familiarization, free exploration of AuslanSpell, and fingerspelling readback tests (multiple-choice and free-text).
  • Metrics: Accuracy in identifying letters and words, usability ratings, and qualitative feedback.
• Limitations and future work:
  • Limited scope to citation forms of letters; future work will explore co-articulation effects and sociolinguistic variation.
  • Testing was restricted to university students; planned studies will include diverse learner groups, such as TAFE students and intermediate signers.
  • Current animations lack stochastic variation; future iterations will integrate generative models for enhanced realism.

Summary

AuslanSpell is a groundbreaking application that leverages motion capture and 3D animation to address the challenges of learning Auslan fingerspelling. By providing interactive features like adjustable speeds and rotatable views, it enhances learner engagement and comprehension. Preliminary user testing with 33 novice signers demonstrated significant learning gains, with participants performing above chance on multiple-choice tasks and correctly identifying first and last letters in free-text tasks. The application is poised to transform Auslan education by offering accessible, individualized learning tools, with future work focusing on expanding datasets, refining animations, and testing with broader learner groups.