ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory

Title of the Paper

ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Augmented Reality (AR), Spatial Memory
• Keywords: Spatial Memory, Augmented Reality, Virtual Extended Display, Hybrid User Interface, Cognitive Load, User Experience, User Study, Display Size, Task Load, Gaze Switching

Research Background and Problem

• Identified Issues or Challenges:

  1. Smartphones face a bottleneck in screen size due to portability and ergonomic optimization, with smaller displays potentially limiting user efficiency, spatial memory, and user experience.
  2. Augmented Reality (AR) devices offer nearly unlimited virtual visual space, but challenges exist in terms of precision and comfort in mid-air interaction.
  3. Whether hybrid user interfaces (e.g., AR-extended smartphone screens) can balance the convenience of smartphone touch interaction with the display space provided by AR requires further investigation.

• Significance of the Research: Exploring how to optimize the combination of smartphones and AR headsets can enhance user efficiency, comfort, and spatial memory in multitasking scenarios, providing critical guidance for future user interface design.

• Motivation and Related Work:

  1. Existing research indicates that larger display spaces positively impact spatial memory, cognitive load, and user experience, but no studies have explicitly explored the optimal virtual screen size in hybrid user interfaces.
  2. Previous studies also suggest that frequent gaze switching in multi-display environments may affect user performance.
  3. While attempts have been made with extended screen technologies (e.g., projection devices, multi-device interaction), the unique structure of AR-extended screens warrants further exploration.

Proposed Solution

• Proposed Method or Solution: The authors propose a hybrid user interface combining smartphones with AR headsets, incorporating a Virtual Extended Screen Attached Display (VESAD). By increasing the virtual screen size, the study aims to optimize user experience, spatial memory, and task load.

• Innovative Aspects:

  1. Introduced four different screen size conditions (no extension, tablet-sized extension, medium desktop monitor-sized extension, and TV-sized extension).
  2. Systematically studied the impact of virtual extended screen size on user spatial memory, cognitive load, and subjective experience, testing the hypothesis of an "optimal display size" for the first time.

• Implementation Steps and Key Technologies:

  1. Designed experimental tasks, including a navigation phase and an object location recall phase, simulating static navigation on a 2D grid map.
  2. Provided four comparative conditions: no virtual extension (NO-AR), small screen virtual extension (SMALL-AR), medium extension (MEDIUM-AR), and large extension (LARGE-AR).
  3. Used video see-through AR technology to avoid parallax issues associated with optical see-through systems.
  4. Implemented the display model using Unity, recording navigation paths, task completion times, eye-tracking, and head movement data.
  5. Each participant completed 24 tasks sequentially and participated in a subjective experience survey and semi-structured interview.

Research Findings

• Specific Results:

  1. Spatial Memory: The medium-sized extended screen (MEDIUM-AR) significantly outperformed other conditions in task completion time, navigation path efficiency, and location recall accuracy. The data suggest that the medium screen represents a "sweet spot" for improving spatial memory.
  2. Task Load:
     • The large extended screen (LARGE-AR) showed slight performance improvements but introduced significantly higher head rotation demands.
     • The small extended screen (SMALL-AR) performed worse than the no-extension condition (NO-AR) in spatial memory, possibly due to visual attention dispersion.
  3. User Experience:
     • MEDIUM-AR and LARGE-AR conditions significantly improved subjective experience ratings, including attractiveness, clarity, and efficiency.
     • MEDIUM-AR achieved the highest average scores, reflecting greater user familiarity and comfort.

• Advantages Compared to Existing Solutions:

  1. Provided the first systematic experimental data on hybrid user interface virtual extended displays.
  2. Identified the specific cognitive impacts of different screen sizes, offering scientific evidence for designing optimal augmented display sizes.

• Experimental or Evaluation Results: The experiment revealed that moderately enlarged virtual extended screens (similar to desktop monitors) significantly reduced task completion time, decreased cognitive load, and enhanced user experience. However, excessively large extensions (similar to TV screens) negatively impacted overall performance due to increased head rotation demands.

• Limitations and Future Directions:

  1. The experimental scenarios for LARGE-AR were relatively conservative; future studies could explore the potential benefits of even larger display spaces.
  2. This study fixed the input modality and did not explore other smart device combinations (e.g., smartwatches or dynamic handheld modes).
  3. The use of video see-through displays requires further validation of real-world interaction effects.

  Future Research Directions:

  • Investigate the optimal apex size ("sweet spot") for virtual extended screens.
  • Compare the effects of curved screen layouts (e.g., CAVE systems) on spatial memory and task load.
  • Examine the effects of visual attention switching between real and virtual environments.
  • Explore user interaction with virtual interfaces beyond the screen, identifying potential use cases and improvements for small extended screens.