Tesseract: Querying Spatial Design Recordings by Manipulating Worlds in Miniature

Document Title

Tesseract: Querying Spatial Design Recordings by Manipulating Worlds in Miniature

Document Information

• Domain: Human-Computer Interaction, Virtual Reality, Spatial Design Record Querying
• Keywords: Spatial design record querying, Worlds-in-Miniature, query tools, virtual reality, multimodal data, temporal navigation, data retrieval, user interaction, design learning

Research Background and Problem

• Problem or Challenge: Spatial design recordings are multimodal data logs that include user activities, workflows, and tool usage. Traditional query tools struggle to efficiently help users locate key moments across multiple recordings. These recordings contain diverse information such as people, objects, and interactions distributed across various time points and sessions.
• Importance: Reviewing spatial design recordings allows designers to reflect on their design processes, learn workflows, and understand others' design intentions. This is crucial for improving design efficiency, reducing redesign costs, and fostering knowledge sharing.
• Research Motivation and Related Work:
  • Previous work primarily focused on navigating single spatial recordings or individual timelines, lacking tools to support cross-record queries.
  • Methods such as direct manipulation of scene objects or timeline replay have functional limitations.
  • Advanced tools are needed to enable designers to express query intentions through multimodal means and retrieve relevant critical design moments.

Solution

• Proposed Method:
  • The Tesseract system is based on the "Worlds-in-Miniature" (WiM) concept, utilizing a "Search Cube" interface and four query tools (object search, proximity search, viewpoint search, and voice search) to support spatial design record querying.
  • Users can create spatial queries using the Search Cube interface, defining objects, human behaviors, and interaction content to retrieve past design activity segments.
• Innovations:
  • Introducing WiM miniature models for querying multimodal spatial design recordings for the first time, combined with four query tools to enable users to express complex design scenarios.
  • Allowing users to combine query tools for hybrid expressions, enhancing query precision and diversity.
  • Supporting cross-record filtering rather than being limited to single recordings or timelines.
• Implementation Steps and Key Techniques:
  • Object Search: Users drag objects into the Search Cube, defining position and weight attributes (presence, absence, importance) and generating queries based on relative layout relationships (e.g., proximity or alignment).
  • Proximity Search: Defining spatial proximity relationships between people and objects within specific time points, visualized through adjustable selection disks.
  • Viewpoint Search: Defining perspectives of individuals or recordings, supporting dynamic movement paths or viewpoint changes described through time frames.
  • Voice Search: Using text embedding methods to match user dialogues with interaction logs and recorded conversations.
  • Retrieved segments are presented as spatial clips, allowing users to preview or jump to 1:1 scale reproductions of design activities.

Research Outcomes

• Specific Results:
  • Tesseract enables designers to query complex design recordings through miniature models and quickly retrieve design activities and key moments from the built-in database.
  • Users can utilize different query tools or combine them to return highly accurate target segments.
• Advantages Over Existing Methods:
  • Compared to traditional temporal navigation, Tesseract supports cross-record queries, multimodal expressions, and searches in complex scenarios, offering greater flexibility to meet user needs.
  • Its query logic allows users to express search intentions through familiar design elements (e.g., objects, human activities) and diverse methods.
• Experiment and Evaluation Results:
  • User feedback indicates that miniature models provide an intuitive query concept, and the query tools are easy to learn and use.
  • Experiments confirm that participants can use Tesseract to retrieve core design moments in spatial design and achieve target segments.
  • Overall evaluations show high acceptance of all query tools, with voice search and object search tools being particularly popular.
• Limitations and Future Directions:
  • Improving query precision: For example, handling object relationships in more complex scenarios (e.g., "on top of," "inside").
  • Problem scale: Optimizing query time, result ranking, and data representation as the number of recordings increases.
  • Supporting more data types: Future expansions could include more interaction data from design software (e.g., color, texture, posture, gestures).
  • Application extensions: Exploring adaptations for larger-scale scenarios (e.g., urban planning) and other spatial design fields (e.g., automotive design, architectural design).

This document provides technical guidance and empirical evaluations for innovative spatial design record querying, representing new advancements in the spatial data field.