Why stressed, Mom?: Exploring Family Reflection on Social and Emotional Sensor Data through Family Informatics

Paper Title

Why stressed, Mom?: Exploring Family Reflection on Social and Emotional Sensor Data through Family Informatics

Publication Info

• Topic area: Family informatics and social-emotional learning through sensor data.
• Keywords: Family informatics, social-emotional learning, relational reflection, multimodal sensing, emotional data, family dynamics, co-reflection, sensor-based systems, HCI, family communication.

Background and Problem

• Problem / challenge: Existing family informatics systems focus on tracking individual or aggregated health data but lack tools to support families in reflecting on their social and emotional interactions. Multimodal sensing of social-emotional states and presenting such data to families remains technically challenging and underexplored.
• Significance: Understanding family interactions through reflection can strengthen relationships, foster children’s social-emotional development, and uncover communal norms. However, the lack of tools for symmetric, balanced reflection in everyday family life limits these benefits.
• Motivation and related work: Prior research has demonstrated the feasibility of using sensor-aided systems for family reflection but has mostly focused on one-way, parent-driven instructional scenarios. There is limited empirical evidence on how sensing can support collaborative reflection in ordinary family settings, particularly for social and emotional dynamics.

Solution

• Proposed approach: SELaD (Social Emotional Learning by multimodal Data), a family informatics system that captures and visualizes social-emotional data from family interactions using audio, video, and physiological sensors.
• Novelty:
  1. Introduced the concept of relational reflection, where families collaboratively interpret multimodal data to understand conversational and emotional dynamics.
  2. Designed and implemented SELaD, integrating multimodal data (audio, video, physiological) into visualizations and prompts for family reflection.
  3. Conducted a semi-naturalistic study with 17 families to investigate how SELaD facilitates reflection and identified factors that deepen the reflective process.
• Procedure and key techniques:
  • SELaD captures multimodal data during family mealtime conversations.
  • Data types include conversation summary (keywords, positive words), conversation behaviors (speaking speed, turn-taking), interaction behaviors (eye contact, emotional synchrony), and physiological responses (stress, emotional arousal).
  • Families use SELaD’s interface to review synchronized video and data visualizations, guided by question-type prompts to facilitate reflection.

Results

• Concrete findings:
  • Families gained insights into conversational patterns (e.g., dominance, turn-taking) and emotional influences (e.g., stress levels, emotional synchrony).
  • Emotional data, particularly its ambiguity and misalignment across sources, catalyzed deeper discussions and perspective-taking.
  • Families engaged in relational reflection by recalling shared histories and expectations, leading to self-evaluation, empathy, and recognition of the need for positive change.
• Advantage over baselines:
  • SELaD enabled collaborative interpretation of multimodal data, unlike prior systems that focused on individual data or one-way feedback.
  • Emotional data facilitated richer, more nuanced reflections compared to straightforward conversational data.
• Experiments / evaluation:
  • Conducted a user study with 17 families (n = 51) in a semi-naturalistic home-like setting.
  • Families engaged in three structured conversation topics (reminiscing, supporting, planning) followed by a reflection session using SELaD.
  • Data analysis included thematic coding of 327 reflection instances and interviews.
• Limitations and future work:
  • Lab-based study may not fully capture real-world family dynamics; future work should involve in-home deployments.
  • Limited generalizability due to cultural and demographic homogeneity (Korean families).
  • Accuracy of emotion recognition models and usability of certain data components (e.g., keyword context) need improvement.
  • Ethical concerns around privacy and power dynamics, especially for children, require further exploration.

Summary

This study introduced SELaD, a family informatics system that uses multimodal sensor data to facilitate relational reflection on social and emotional dynamics in family interactions. Through a user study with 17 families, the research demonstrated how SELaD enabled families to collaboratively interpret data, uncover conversational patterns, and explore emotional influences. Emotional data, particularly its ambiguity and misalignment, played a key role in deepening reflection. The findings highlight the potential of multimodal sensing to support family sensemaking and suggest design implications for contestable, symmetric, and privacy-conscious systems. Future work should address technical, usability, and ethical challenges to enhance the applicability of such systems in diverse family contexts.