ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Title of the Paper

ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Paper Information

• Domain: Human-Computer Interaction (HCI) and Video Conferencing Technology
• Keywords: Video communication, object sharing, collaborative work, enhanced communication, task space, digital objects

Research Background and Problem Statement

• Problems and Challenges:

  • Demonstrating physical objects in video meetings faces limitations such as screen space constraints and camera angle issues. Users often need to repeatedly adjust the position of objects and the camera to showcase details, maintain visibility, and reference specific parts, making the process cumbersome.
  • Virtual backgrounds may obscure parts of objects, making it difficult to display them effectively.
  • Existing solutions require complex hardware setups, which are not suitable for everyday use scenarios.

• Significance: With the increase in remote work, education, and social activities, enhancing the ability to showcase physical objects in video meetings is crucial for improving collaboration efficiency and user experience.

• Research Motivation and Related Work:

  • Existing literature focuses on hardware extensions (e.g., multiple cameras) or sharing fixed task spaces but lacks in-depth research on achieving seamless object sharing using everyday devices.
  • The task space in video conferencing lacks privacy protection for users or the ability to emphasize object details. This study aims to fill this design gap.

Solution

• Method or Solution:

  • Proposes a video conferencing system called ThingShare, which uses instance segmentation technology to digitize physical objects, allowing users to easily create and manipulate digital objects. It supports ad-hoc interactions and detailed task space discussions.
  • ThingShare provides two view modes:
    • Person View: Combines the user's personal video with the digitized object, integrating it with the user's presentation.
    • Task View: Displays magnified object details, supports annotation features, and enables multi-perspective discussions.

• Innovations:

  • ThingShare employs a software-based solution that achieves real-time digitization of physical objects without requiring additional hardware.
  • Users can flexibly freeze, store, and manipulate objects as digital copies, while supporting collaborative referencing of digital objects among multiple users.
  • Provides privacy protection in virtual backgrounds and enhances object views.

• Specific Techniques and Implementation Steps:

  • Extracts objects from user videos and runs an instance segmentation model (Yolact), combined with image processing techniques (e.g., morphological operations) to enhance object detection.
  • Users can operate on objects through drag-and-drop and context menus, including creating snapshots, short videos, and live videos.
  • Offers an interactive user interface that allows users to switch between real-time views and an object library, supporting multi-angle discussions in shared task spaces.

Research Outcomes

• Specific Outcomes:

  • ThingShare significantly improves the efficiency of showcasing and referencing physical objects in video meetings.
  • Users can seamlessly operate objects from physical to digital spaces and collaborate through the Task View.
  • The system resolves the cumbersome process of showcasing details with traditional methods and supports diverse use cases such as social sharing, collaborative design, and remote assistance.

• Advantages:

  • Requires no complex hardware and enables convenient online collaboration among multiple users.
  • Supports object sharing in virtual backgrounds while protecting user privacy.

• Experimental or Evaluation Results:

  • User studies indicate that ThingShare outperforms traditional video conferencing software in showcasing object details, supporting multi-perspective discussions, and maintaining privacy.
  • Most users preferred ThingShare's non-real-time object digitization features (e.g., snapshots and short videos).

• Limitations and Future Directions:

  • Object detection and segmentation models need further optimization for multi-angle and specific scenarios.
  • The current design is based on dyadic interactions; future work should expand to multi-party or more complex collaborative scenarios.
  • Support for post-editing or downloading captured objects is needed to accommodate practical applications.