Understanding Spatiotemporal-Aware Multimodal Conversational Search in the Outdoor Urban Space

Paper Title

Understanding Spatiotemporal-Aware Multimodal Conversational Search in the Outdoor Urban Space

Publication Info

• Topic area: Multimodal conversational search in dynamic urban environments.
• Keywords: Multimodal conversational search, spatiotemporal awareness, urban mobility, human-computer interaction, mobile information seeking, contextual inquiry, AI ethics, spatial reasoning, user experience, privacy.

Background and Problem

• Problem / challenge: Existing conversational search tools focus on stationary, indoor contexts and lack robust integration of spatiotemporal awareness. There is limited empirical understanding of how users interact with such tools in outdoor urban spaces, where dynamic surroundings and fragmented attention create unique challenges.
• Significance: Addressing these gaps can improve real-time information seeking, decision-making, and navigation in urban environments, enhancing user experience and safety.
• Motivation and related work: Prior research has explored conversational search and AI systems in urban spaces but has not sufficiently addressed spatiotemporal-aware interactions or their implications for user behavior, needs, and concerns in outdoor settings.

Solution

• Proposed approach: Development of UrbanSearch, a multimodal conversational search (MCS) tool that integrates spatiotemporal awareness (e.g., geolocation, timestamp, visual surroundings) to support real-time information seeking in urban spaces.
• Novelty:
  1. Empirical insights into user interactions with spatiotemporal-aware MCS tools in outdoor urban contexts.
  2. Identification of unmet needs, such as spatial reasoning and transparent multi-source integration.
  3. Design implications for future MCS systems, including compact multi-turn conversations and visualized spatial reasoning.
  4. Exploration of user concerns regarding peripheral awareness, social interaction, and privacy.
• Procedure and key techniques:
  1. Developed UrbanSearch, which combines vision-language models, GPS, and web search capabilities.
  2. Conducted a contextual inquiry with 23 participants in Tampere, Finland, involving walking sessions and semi-structured interviews.
  3. Analyzed user interactions, query patterns, and feedback using thematic analysis and triangulated data sources (probe logs, screen recordings, interview transcripts).

Results

• Concrete findings:
  • Participants submitted an average of 26.2 queries and 19.0 images per session, with UrbanSearch achieving 94.7% response accuracy.
  • Interaction threads followed an Observe–Decide–Act flow, with most threads ending at the Observe or Decide stages.
  • Participants valued low-effort query formulation and highly relevant, context-aware responses.
• Advantage over baselines:
  • UrbanSearch reduced cognitive and manual effort by leveraging shared spatiotemporal context, outperforming traditional search engines in relevance and usability for on-the-move queries.
  • Functioned as a central information gateway, reducing the need to switch between multiple apps.
• Experiments / evaluation:
  • Conducted with 23 participants (diverse demographics and AI familiarity) in a real-world urban setting.
  • Evaluated response accuracy and user trust through qualitative and quantitative methods.
• Limitations and future work:
  • Single-session study in one European city; future work should include longitudinal studies and cross-cultural contexts.
  • Limited exploration of emerging devices like smart glasses.
  • Need for further research on user trust calibration and taxonomy of effort reduction.

Summary

This study investigated spatiotemporal-aware multimodal conversational search (MCS) in urban spaces using UrbanSearch, a technology probe integrating GPS, timestamp, and visual inputs. Findings revealed that MCS tools reduce cognitive and manual effort, support environmental learning, in-situ decision-making, and personalized navigation, and serve as central information gateways. However, unmet needs for spatial reasoning, multi-source transparency, and concerns about privacy and social awareness were identified. The study proposes design implications for compact multi-turn conversations, spatial reasoning visualization, and external tool integration, paving the way for more adaptive, context-aware, and socially sensitive MCS systems.