ASL Sea Battle: Gamifying Sign Language Data Collection

Title of the Paper

ASL Sea Battle: Gamifying Sign Language Data Collection

Paper Information

• Research Area: Sign Language Data Collection and Machine Learning
• Keywords: Sign Language Recognition, ASL (American Sign Language), Gamification, Data Annotation, Board Games, Human-Computer Interaction, Accessibility, User Research

Research Background and Problem Statement

• Identified Problem: Current sign language recognition and translation systems struggle to perform reliably in real-world environments, primarily due to the lack of high-quality training data. Existing datasets are generally small in scale, lack diversity, are not representative of real-world scenarios, and often contain missing or erroneous labels.
• Importance: Accurate machine learning models for sign language can open up channels of information and communication for millions of sign language users, enabling applications such as digital assistants, automatic sign language transcription services, and sign language-to-speech translation.
• Motivation and Related Work:
  • Existing resources, such as sign language dictionaries, educational materials, and databases, are mostly non-commercial and offer limited functionality, lacking entertainment and interactivity.
  • Gamified data collection has been successfully applied in other fields (e.g., protein folding, image annotation), but no dedicated games have yet been developed for sign language users.

Solution

Methodology and Innovation

• The authors propose ASL Sea Battle, a mobile game based on the classic board game "Battleship," where players use sign language strategies to attack game grids, thereby collecting real-world sign language videos and their labels. This approach integrates data collection and annotation while providing users with entertainment and educational experiences.
• Innovations in the game design:
  1. Each grid cell is associated with a specific sign language label, and players attack by recording corresponding sign language videos.
  2. Opponents verify the videos and associated grid labels to complete annotation tasks.
  3. The game introduces bidirectional interaction and competitive mechanics to enhance user engagement.

Implementation Steps and Technology

1. Design Process:
   • The authors drew inspiration from multiple existing board game prototypes and conducted iterative testing to select the design that best met their needs (e.g., Scattergories, Hangman, and Battleship).
   • Fluent sign language users, particularly members of the Deaf community, were actively involved in the design and debugging phases, becoming integral members of the research team.
2. Data Collection and Annotation Workflow:
   • Players record sign language videos in real-world scenarios to execute attack commands.
   • Videos are automatically sent to opponents, who complete the annotation tasks and unlock grid cells.
3. Technical Implementation:
   • The game was developed using React Native and supports both Android and iOS platforms.
   • The backend employs Node.js with the Express framework, ensuring game state synchronization via REST APIs and MongoDB.
   • Azure cloud services were used for data storage, with standard HTTPS security protocols for deployment.

Research Outcomes

Data Quality and User Experience

1. Data Quality:
   • A total of 657 videos were collected during user research, with 98% evaluated by experts as meeting the requirements for training models.
   • Label accuracy reached 100%, indicating that users could reliably complete annotation tasks, even when some users were not fluent in sign language.
2. Data Throughput:
   • The interactive workflow of ASL Sea Battle resulted in a slightly slower data collection rate compared to traditional methods but provided additional entertainment value.
   • The average recording time was 17.57 seconds, and label recognition time was significantly influenced by the gamified design.
3. Engagement and Preferences:
   • 95% of participants found ASL Sea Battle more enjoyable; 85% felt the gamified design increased their willingness to use the game for extended periods.
   • Deaf users particularly valued the game's ability to connect communities and its combination of entertainment and education.

Limitations and Future Directions

1. Limitations:
   • The game experiment was limited to a small set of basic sign language vocabulary; future work needs to expand to datasets with more complex or continuous sign language content.
   • Reference videos within the game may vary based on regional dialects or personal habits, potentially causing confusion for some fluent users.
   • Privacy concerns regarding sign language videos need to be addressed for real-world deployment.
2. Future Plans:
   • Long-term deployment to collect a large-scale dataset sufficient for model training.
   • Development of sign language recognition models capable of handling low-quality data in natural scenarios.
   • Exploration of applications in social, educational, and professional contexts for sign language games.
   • Expansion to other sign language communities, such as users of International Sign Language.

Conclusion

ASL Sea Battle leverages gamification to address challenges in sign language data collection and annotation, advancing the development of sign language technologies. The study demonstrates that designs combining entertainment, education, and social interaction can enhance user engagement while providing high-quality data for model training. This approach could become a scalable method for creating sign language technologies and offer new resources and services for the Deaf community and sign language learners.