Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign Recognition

Title of the Paper

Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign Recognition

Paper Information

• Subject Area: American Sign Language (ASL) learning and translation technology, hybrid search interface design, and user experience research
• Keywords: sign language, American Sign Language (ASL), dictionary, search interface, video search, user satisfaction, information retrieval effectiveness, search evaluation, search system design

Research Background and Problem

• Problem or Challenge:

  • Learners face difficulties when using sign language to English dictionaries to look up unfamiliar sign language vocabulary, such as the inability to query via definitions.
  • Current video recognition technology cannot fully and accurately recognize complex 3D gestures, resulting in lengthy result lists that require significant effort to locate the correct vocabulary.
  • Traditional feature-based dictionary search demands a high level of knowledge from sign language learners, particularly beginners who often struggle to recall precise linguistic features.

• Significance:

  • With the growing number of American Sign Language learners (approximately 200,000), designing more efficient sign language translation tools is crucial for fostering communication and social inclusion between the Deaf community and hearing individuals.

• Research Motivation and Related Work:

  • The rise of video query-based search systems offers new possibilities for sign language dictionaries but still faces challenges with inaccurate recognition results.
  • No prior research has focused on designing a "hybrid search" method for sign language dictionaries, which combines video queries with feature-based filtering to address these challenges.

Solution

• Proposed Method:

  • A hybrid search system combining video queries with language feature-based filtering functionality.
  • Users first submit a video to generate a list of potential matches, then narrow down results using a filtering interface based on semantic features (e.g., handshape, location).

• Innovative Aspects:

  • Introduction of post-query result filtering to enhance the user experience in handling lengthy result lists and reduce reliance on precise search inputs.
  • Focus on improving the usability of video search rather than enhancing AI recognition performance.

• Implementation Steps and Key Technologies:

  • Investigate beginner users' needs for filtering interfaces, result page content, and presentation.
  • Design a Wizard-of-Oz prototype based on the research and evaluate its performance.
  • Learn user preferences to optimize filtering functions and the design of text or image labels.

Research Outcomes

• Specific Findings:

  • Provided user preference data for hybrid search interfaces and result page designs, such as automatic video playback and multiple linguistic feature filtering options.
  • Users expressed significantly higher satisfaction with hybrid search systems that included filtering functionality compared to video-only search systems.
  • Hybrid search users felt more in control of the search process, with a slight improvement in task success rates.

• Advantages in Comparison:

  • Hybrid search is easier to navigate and locate target vocabulary than traditional video search, especially when result lists are lengthy.
  • Significantly improved user satisfaction and reduced search abandonment rates without requiring higher video recognition accuracy.

• Experimental or Evaluation Results:

  • The average satisfaction of hybrid search users was higher than that of video search users.
  • The proportion of users finding the correct vocabulary using hybrid search was 84%, compared to 79% for the video search group.
  • Hybrid search demonstrated faster search speeds for results near the top but had slightly longer overall search times.

• Limitations and Future Directions:

  • Limitations:
    • This study only tested the prototype system on desktop browsers, excluding usage scenarios on mobile or other devices.
    • Experimental tasks involved searching for single sign language vocabulary items, without addressing long texts, historical memory, or language learning contexts.
  • Future Directions:
    • Explore broader user groups, such as younger learners not enrolled in formal sign language courses and Deaf users.
    • Research layout optimization of filtering interfaces and their impact on user selection behavior.
    • Conduct longitudinal studies to evaluate the educational benefits of hybrid search.

Conclusion

This study designed and experimentally validated a hybrid search method combining video and feature-based filtering for sign language dictionaries, offering a series of design optimization suggestions and expanding the application prospects of related search technologies in language learning and motion recognition. The research not only provides important references for the future design of sign language learning tools but also inspires interest in hybrid methods for other language or video search systems.