Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners

Paper Title

Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners

Publication Info

• Topic area: Design and development of socially intelligent AI systems with Theory of Mind capabilities.
• Keywords: Theory of Mind, socially intelligent AI, human-AI interaction, dynamic mental states, subjective AI design, situated AI, co-design, AI practitioners, user-facing AI.

Background and Problem

• Problem / challenge: Current AI systems lack the ability to infer and respond to transient, implicit human mental states, limiting their effectiveness in everyday social interactions. Existing approaches to AI Theory of Mind (ToM) are largely static, inference-driven, and disembodied, which may not align with real-world user-facing applications.
• Significance: Enhancing AI with ToM capabilities could improve user experience by enabling systems to adapt to dynamic, context-sensitive, and subjective human needs, fostering more meaningful human-AI interactions.
• Motivation and related work: Prior research has focused on computational modeling and benchmarking ToM capabilities in AI, but little attention has been given to how ToM-enabled AI could be designed for everyday contexts. Related work on socially intelligent AI highlights the need for systems to adapt to users’ evolving needs and social environments, but challenges such as privacy concerns and rigid design paradigms remain unresolved.

Solution

• Proposed approach: Co-design sessions with AI practitioners to envision and reflect on the design of ToM-enabled AI products and services that are situated, dynamic, and subjective.
• Novelty:
  1. Identification of three design recommendations for ToM-enabled AI: situated in social context, responsive to dynamic mental states, and attuned to subjective individual differences.
  2. Exploration of design tensions between envisioned ToM-enabled AI futures and current AI design practices.
  3. Proposal of a design direction treating ToM as a pervasive capability embedded within AI functionalities, supporting continuous human-AI interaction loops.
• Procedure and key techniques:
  • Conducted 13 virtual co-design sessions with 26 U.S.-based AI practitioners from engineering and design roles.
  • Practitioners engaged in learning about ToM, designing ToM-enabled AI features for six human-AI social misalignment scenarios, and reflecting on real-world challenges and opportunities.
  • Data analyzed using affinity diagramming and reflexive thematic analysis to distill design recommendations and tensions.

Results

• Concrete findings:
  • ToM-enabled AI should:
    1. Be situated in social contexts, using multi-modal data sources (e.g., visual cues, voice, environmental data) and unobtrusive infrastructure.
    2. Support dynamic mental states through continuous monitoring and adaptive responses.
    3. Attune to subjective individual differences, accommodating nuanced and personal mental states.
• Advantage over baselines: Proposed ToM-enabled AI designs move beyond static, inference-driven approaches by emphasizing context-sensitive, adaptive, and personalized interactions, addressing limitations of current AI systems.
• Experiments / evaluation:
  • Practitioners designed speculative AI features for scenarios such as smart home assistants, autonomous vehicles, and AI cooking robots.
  • Reflections revealed challenges in balancing user privacy, computational feasibility, and subjective personalization.
• Limitations and future work:
  • Study grounded in classical inference-based ToM definitions; alternative perspectives (e.g., embodied cognition) could yield different insights.
  • Scenarios focused on U.S.-based consumer contexts; broader cultural and contextual diversity needed.
  • Applicability to professional or high-stakes settings (e.g., healthcare, business) remains unexplored.

Summary

This study explored the design of Theory-of-Mind-enabled AI systems through co-design sessions with industry practitioners, identifying three key recommendations: situatedness in social contexts, responsiveness to dynamic mental states, and attunement to subjective individual differences. Findings highlight tensions between envisioned futures and current AI design practices, such as privacy concerns and scalability challenges. The paper proposes treating ToM as a pervasive capability embedded within AI functionalities, enabling continuous interaction loops to enhance user experience. Future work should expand cultural contexts, explore alternative ToM frameworks, and investigate applications in high-stakes domains.