Selecting Real-World Objects via User-Perspective Phone Occlusion

Document Title

Selecting Real-World Objects via User-Perspective Phone Occlusion

Document Information

• Subject Area: Human-Computer Interaction, physical object selection via smartphones
• Keywords: object selection, smartphone interaction, user perspective, ROI selection, spatial interaction, camera occlusion, visual feedback techniques, target selection, interaction efficiency

Research Background and Issues

• Identified Problems or Challenges:

  1. Traditional smartphone-based methods (e.g., selecting targets via camera preview displayed on the screen) often require multiple steps, leading to inefficiency.
  2. Some existing methods directly use the device (e.g., the direction of the phone camera) for target selection, but the lack of visual feedback results in poor accuracy and potential user discomfort.
  3. In dense scenes, traditional beam-based or user-perspective interaction methods may struggle to precisely locate targets, especially in overlapping or crowded object scenarios.

• Significance of the Problem: Smartphones are essential tools for daily interaction, and enabling them to quickly and accurately select objects of interest is crucial for enhancing user experience and interaction efficiency. Addressing issues of insufficient visual feedback that lead to user discomfort and reduced usability is also a key significance of this research.

• Research Motivation and Related Work:

  1. Existing user-perspective-based interactions often face the "double-vision problem," where users find it difficult to confidently select the correct target.
  2. Designing a simple and direct user interaction method can improve real-time target selection and efficiency.
  3. Research on object selection in mixed virtual and real-world scenarios provides design insights for future IoT interactions.

Proposed Solution

• Proposed Method: The authors propose an innovative target selection technique that uses the occlusion area of a smartphone for target selection. This approach transforms the phone into a physical cursor for interacting with the real world, based on the user’s perspective, by providing a scalable and rotatable ROI (Region of Interest) for target selection.

• Innovations:

  1. Eliminates the constraints of camera preview by enabling users to confidently select targets through occlusion-based visual feedback.
  2. Supports target selection and disambiguation in dense scenes using a "rectangular region cursor" approach.
  3. Combines the front and rear cameras of the smartphone to capture the user’s eye position and occlusion area, while analyzing user behavior models to improve target prediction accuracy.

• Implementation Steps:

  1. Occlusion Area Estimation: Detect the user’s iris position using the MediaPipe Iris algorithm via the front camera and estimate the occlusion area in the rear camera image based on the phone’s fixed characteristics.
  2. Object Detection: Use the YOLOv4 deep learning model to detect interactive objects in the rear camera image.
  3. Target Prediction Algorithm: Calculate target selection probability based on the distance-weighted Jaccard index between the occlusion rectangle and target objects, combined with user behavior models for precise matching.
  4. Prototype Development: Implement the prototype on an iPhone 12 Pro, utilizing cloud computing for processing and evaluating performance and accuracy.

Research Outcomes

• Specific Results:

  1. The proposed occlusion area algorithm achieved an accuracy of 1.28°±0.96°, which can be further improved to 0.65°±0.52° with user calibration.
  2. User studies demonstrated that the occlusion selection technique significantly outperformed traditional methods in terms of efficiency, accuracy, and user acceptance.
  3. Comparative analysis of multiple candidate methods showed that occlusion selection resulted in lower task load (NASA-TLX) and higher system usability (SUS).

• Advantages:

  1. Adapts to an intuitive user-perspective interaction model, addressing the discomfort caused by insufficient visual feedback in traditional methods.
  2. Simplifies the target selection process, enhancing the immediacy and coherence of interactions.
  3. Ensures accurate selection results in complex, dense scenes through target disambiguation.

• Experimental and Evaluation Results:

  1. Two user studies validated the accuracy of the occlusion area estimation algorithm and the user experience, showing that the occlusion selection technique significantly improves target selection efficiency.
  2. Many participants in the user survey preferred the occlusion technique, praising its simplicity and comfort.

• Limitations and Future Directions:

  1. Currently, the system can only detect predefined object sets; future work should enable dynamic registration of new objects.
  2. For complex-shaped objects, more precise segmentation models are needed to improve the algorithm.
  3. To address current depth estimation errors, short-range depth sensors (e.g., LiDAR) on smartphones could further optimize the system.
  4. Expand the application of the technique to other scenarios (e.g., AR/VR) to address the double-vision problem and enhance user experience.