Show of Hands: Leveraging Hand Gestural Cues in Virtual Meetings for Intelligent Impromptu Polling Interactions

Title of the Paper

Show of Hands: Leveraging Hand Gestural Cues in Virtual Meetings for Intelligent Impromptu Polling Interactions

Paper Information

• Research Domain: Human-Computer Interaction, Virtual Meeting Technologies, Gesture Recognition
• Keywords: Gesture Recognition, Virtual Meetings, Impromptu Polling, Interactive Interface, Augmented Reality, User Studies, Educational Technology, Task Workload, Visual Feedback

Research Background and Problem

• Identified Issues:

  • Virtual meetings lack the flexibility of face-to-face meetings in utilizing non-verbal signals such as gestures and facial expressions for interaction, particularly for impromptu polling functions.
  • Existing polling tools (e.g., Poll Everywhere and Kahoot!) require prior preparation, making them unsuitable for spontaneous questioning scenarios.
  • Common video conferencing platforms (e.g., Zoom and Google Meet) offer features like emojis, but hosts must manually review feedback, which significantly increases cognitive load as the number of participants grows.
  • Compared to traditional face-to-face meetings, virtual meetings lack elements that foster interaction and rapport through visual and auditory cues.

• Significance:

  • Virtual meetings have become the norm in modern education and professional fields, yet improving engagement and interactivity remains a pressing challenge.
  • Actively sensing participant states can not only enhance meeting outcomes but also make participants feel acknowledged, thereby increasing their sense of involvement.

• Research Motivation and Related Work:

  • By integrating artificial intelligence and machine learning technologies, it is possible to design intelligent interactive interfaces capable of capturing natural gestures. Gesture recognition has been applied to symbolic language transmission, non-verbal cue detection, and the development of natural input mechanisms.
  • This work focuses on gesture recognition and explores its application in virtual meeting polling scenarios to alleviate the cognitive burden on hosts.

Solution

• Proposed Method:

  • Design an intelligent user interface that supports impromptu polling interactions, leveraging real-time gesture recognition and video-based visual feedback.
  • Participants perform natural gestures via their cameras, which are automatically recognized by the system and aggregated for feedback. Hosts can then understand meeting dynamics through visualized summary data.

• Innovations:

  • Conducting open-ended gesture elicitation studies to explore participants' naturally chosen gesture sets, ensuring gestures are intuitive and easy to learn.
  • Introducing various visual feedback mechanisms (e.g., color filters, emojis, pop-up result displays) to help hosts assess overall responses.
  • Implementing real-time camera-based gesture detection and result aggregation to provide practical solutions for efficient polling interactions.

• Implementation Steps:

  1. Gesture Elicitation Study: Conduct experiments to determine users' intuitive gesture preferences for binary questions, multiple-choice questions, and scale-based questions.
  2. Visual Feedback Study: Investigate which feedback formats best help hosts quickly understand participant responses.
  3. System Implementation and Evaluation:
     • Develop real-time gesture recognition using Snap Camera and color filters.
     • Build tools for aggregating and visualizing polling results to assist hosts in quickly interpreting statistical data.
     • Validate system accuracy and usability through experiments evaluating gesture detection and host cognitive workload.

Research Outcomes

• Specific Results:

  • The gesture elicitation study identified a set of highly consistent gesture options, such as raising hands (indicating selection), finger counting (for multiple-choice questions), and virtual scale gestures (for scale-based questions).
  • Visual feedback studies revealed that color filters are the most intuitive and preferred method, while emojis and pop-up feedback also have appeal.
  • System experiments demonstrated a gesture detection accuracy of up to 95%, supporting diverse polling scenarios.
  • NASA TLX workload assessments showed that the gesture recognition system significantly reduced hosts' cognitive load.

• Advantages Compared to Existing Solutions:

  • The system eliminates the need for hosts to manually tally participant feedback, reducing cognitive burden in large-scale meetings.
  • The combination of gestures and visual feedback enhances the system's natural interactivity and user experience.

• Limitations and Future Directions:

  • Limitations:

    • The system requires participants to turn on their cameras, which may be restricted in certain anonymous scenarios.
    • Current research tests are limited to educational settings and a narrow range of question types.
    • Real-time performance and scalability for large-scale meetings require further exploration.

  • Future Directions:

    1. Develop a broader gesture set to cover more question types.
    2. Enhance support for anonymous user interactions, such as using dynamic avatars instead of real video feedback.
    3. Expand the system to other meeting scenarios (e.g., professional seminars) and larger participant scales.
    4. Improve the intelligence of automated polling counts to accommodate asynchronous multi-gesture scenarios.
    5. Optimize multi-platform compatibility, enabling participation from mobile device users and those with low-end hardware.

Conclusion

This paper introduces a gesture recognition and intelligent user interface technology to provide an intuitive and flexible impromptu polling solution for virtual meetings. The system not only improves hosts' management efficiency but also enhances participant interaction experiences. Future work will focus on expanding the system's applicability while further optimizing user experience and technical performance.