CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual Reality

Research Background and Problem

• What problems or challenges did the authors identify?
  Although generative AI can already generate and modify 3D objects, most existing methods adopt a "input-to-output" approach, lacking deep, real-time collaboration between users and AI. Furthermore, the representation of AI's intentions during the co-creation process of 3D objects remains unclear, which may lead to users being unable to predict or understand AI behavior, thereby affecting collaboration experience and outcomes.

• Why is this problem important?
  Effectively leveraging generative AI for co-creation in virtual reality (VR) environments can enhance creation quality, user engagement, and satisfaction. If AI's decision-making process and contributions are not comprehensible, users may lose a sense of control over the creative process, undermining trust in the collaboration.

• Research Motivation and Related Work
  Inspired by human collaboration, the authors propose improving user-AI interaction quality through different AI representation modes (such as embodied avatars, incremental visualization of AI changes, and highlighting modified areas). These modes draw on related work in 3D modeling, collaborative creation systems, and embodied AI avatars.

Solution

• What methods or solutions did the authors propose?
  The authors designed a "Wizard-of-Oz" experiment to study the impact of three AI representation modes on the experience of co-creating 3D objects: embodiment, incremental visualization, and highlighting modification areas. Users collaborated with AI in a virtual environment to complete 3D modeling tasks using these modes.

• What are the innovative aspects of this solution?

  1. Focus on the representation modes of AI-user interaction rather than the generative technology itself.
  2. Use VR to create an immersive co-creation environment, allowing users to intuitively experience AI's role.
  3. Examine the impact of these three modes on various aspects of user perception (e.g., support, efficiency, creativity, partnership, system appeal) and gain deep insights through quantitative data and qualitative analysis.

• What are the implementation steps and key technologies used?

  1. Experimental Setup: Users stood in the center of a virtual room and collaborated with AI to modify four types of initial 3D objects (including dinosaurs, chairs, etc.). The AI was implemented using the "Wizard-of-Oz" approach, where pre-designed 3D artists made modifications instead of real generative AI outputs.
  2. Representation Mode Settings: (1) Whether an AI avatar was present (embodiment or not); (2) Whether modifications were displayed incrementally; (3) Whether modification areas were highlighted.
  3. Data Collection: Record user interaction behaviors, drawing time, and subjective feedback, supplemented by follow-up interviews to reveal deeper emotions and behavioral motivations during the experience.
  4. Data Analysis: Use quantitative tools (e.g., multifactor analysis of variance) and qualitative methods (thematic analysis) to process experimental results.

Research Findings

• What specific findings were obtained?

  1. Highlighting
     Highlighting did not significantly improve the predictability or clarity of AI behavior; instead, it reduced participants' enjoyment of collaboration and satisfaction with the completed models.
  2. Incremental Visualization
     Incremental visualization increased users' attention to AI actions but did not show significant improvements in efficiency or trust. Unexpectedly, incremental visualization heightened the perception of "surprise" in AI modifications.
  3. Embodiment
     AI embodiment enhanced users' perception of AI's supportiveness, companionship, and partnership during collaboration, but simultaneously reduced users' sense of ownership over the final creation.

• What advantages does it have compared to existing solutions?
  This study provides a nuanced understanding of the impact of AI representation modes, which are often overlooked and rarely comprehensively compared. Additionally, it goes beyond focusing solely on AI technology itself, emphasizing holistic improvements in user experience design.

• What were the experimental or evaluation results?
  Quantitative Results:

  • User interaction behaviors significantly decreased when highlighting was combined with AI embodiment.
  • Users spent more time drawing when incremental visualization was absent.
  • With AI embodiment, most users perceived AI as a more creative and supportive collaborator.
    Interview Results:
  • Some users expressed a desire to customize the appearance of the AI avatar.
  • Users were dissatisfied with the slow pace of AI actions but appreciated its ability to enhance collaboration.

• Limitations and Future Directions

  1. Limitations:
     • AI behavior was pre-designed and not based on real generative processes.
     • Users could only modify a limited range of objects, potentially restricting creative freedom and the full potential of AI.
     • The sample size was small, with only 16 participants.
  2. Future Directions:
     • Explore AI representation modes in more complex environments (e.g., 3D collaborative modeling in shared environments).
     • Investigate the potential impact of customizing AI avatar appearances on user perception of collaboration.
     • Examine the importance of efficiency and response time in larger-scale generative systems.

Conclusion

This study provides design insights into AI representation modes in VR environments:

1. Avoid combining highlighting and AI embodiment, as it reduces overall user satisfaction.
2. Timeliness in visualizing processes is critical unless the creation process itself is central to the task.
3. AI embodiment can enhance users' sense of support and collaboration but requires careful consideration of its potential impact on users' sense of ownership.

With these insights, future designers can better create effective 3D co-creation systems that collaborate seamlessly with humans.