A picture is worth a thousand words? Investigating the Impact of Image Aids in AR on Memory Recall for Everyday Tasks

DOI

Memory augmentation has long been a central field in Human-Computer Interaction (HCI) research. Recently, emerging multimodal large language models (MLLMs) have extended research on memory augmentation by enabling the retrieval of information stored in multiple formats (e.g., text and image) through free-form queries. However, literature has focused on text-based memory aids, there has been surprisingly limited research on image-based assistance, despite humans' superior efficiency in processing visual information. Therefore, in this work, we explore the effect of image aids on memory augmentation. To this end, we first design and implement an augmented reality (AR) memory augmentation system, informed by human evaluation of MLLM performance (GPT-4o, LLaVA, and Mini-Gemini) and insights from user interface (UI) design workshops. As a result, we found that GPT-4o is most suitable for our system, images complemented with text (i.e., Image+text) are the most preferred format of memory aids. We also identified optimal UI design parameters for AR-based memory augmentation. With a finalized version of the system prototype, we conduct a user study (N=20) consisting of two tasks that simulate real-life memory-related challenges. We found that \textit{Image+text} significantly enhanced both recall performance and memory vividness. Additionally, from a user experience perspective, Image+text was considered the most helpful and easiest to use for memory augmentation. Our findings showed that images are a powerful modality for enhancing memory recall, extending beyond traditional text-based approaches. We expect that insights gained from this work will contribute to the development of practical, everyday memory augmentation systems.