ReMirrorFugue: Examining the Emotional Experience of Presence and (Illusory) Communications Across Time

Research Background and Issues

• What problems or challenges did the authors identify?
  The authors explored how technology can enable cross-temporal "telepresence," allowing users to feel the presence of the past and establish emotional connections with it. This temporal extension of telepresence includes interactions with deceased individuals or one's past self. However, empirical studies in this domain are scarce, and many aspects of designing the technological mechanisms for such emotional interactions remain unknown.

• Why is this issue important?
  Establishing emotional connections with the past not only helps individuals better recall memories but also fosters self-reflection, which is significant for mental health and emotional relief. Moreover, connecting the past and present through technology holds substantial value for sustaining emotional bonds in family, friendships, and self-relationships. It also opens new avenues for designing "commemoration and continuity technologies" aimed at coping with the loss of loved ones or major life transitions.

• Research motivation and related work
  Existing research primarily focuses on spatial remote interaction through telepresence technology rather than cross-temporal interaction. Additionally, current designs often avoid overly realistic visuals to prevent the "uncanny valley effect." This study explores how to achieve cross-temporal telepresence, including interactions with deceased individuals, aiming to enhance users' connections with the past.

Solution

• What methods or solutions did the authors propose?
  The authors designed an augmented piano system called MirrorFugue, which synchronizes the projection of a pianist's hands and body with the mechanical movements of piano keys. This system simulates the telepresence of past piano performances and investigates its impact on users' emotional experiences.

• What are the innovative aspects of this solution?

  • Cross-temporal telepresence: The study extends telepresence interaction from spatial to temporal dimensions.
  • Multisensory interactive experience: MirrorFugue combines real-time piano sound playback, synchronized mechanical key movements, and full-size projections to create an immersive experience.
  • Participatory interaction: Users can choose to "improvise" and interact with recorded past pianists, enhancing their sense of active participation.

• What are the implementation steps and key technologies used?

  • Use of a "player piano" equipped with synchronized sound playback functionality.
  • Projection of past piano performances, ensuring alignment between the projection and piano key movements.
  • Data collection through mixed methods, including physiological signals (heart rate and skin conductance), questionnaires, and interviews, to explore users' multidimensional emotional experiences.

Research Outcomes

• What specific outcomes were achieved?

  • Establishment of telepresence and emotional connection: The majority of participants reported increased emotional connection with the pianist after the experiment, even if the pianist was deceased or unrelated to them.
  • Illusion of time travel: Many participants felt a cross-temporal connection during the interaction, with some describing it as "time travel," such as "the past coming to the present" or "the present going to the past."
  • Positive impact of participatory interaction: Improvisation allowed participants to feel a simulated two-way exchange with the recorded pianist, akin to an "imaginary dialogue."

• What advantages does it have compared to existing solutions?

  • Interaction with deceased individuals: Compared to existing commemorative design frameworks, MirrorFugue offers higher immersion and interactivity, moving beyond simple remembrance.
  • Strengthening emotional bonds: It provides a space for reconnecting with fragments of loss, fostering continuity in social relationships and one's past self.
  • Broad adaptability: Suitable for various relationships in life, including interactions with strangers, family members, and one's past self.

• What were the experimental or evaluation results?

  • Statistical data: Interaction with deceased pianists elicited significantly stronger heart rate changes and emotional responses, indicating that such interactions may trigger deep emotional connections.
  • Interview feedback: Participants reported that the experience evoked numerous memories, including shared moments with the pianist, past personal struggles, and life transitions.

• Limitations and future directions

  • Long-term benefits of interaction remain unexplored: The study has not yet conducted longitudinal investigations to verify the system's sustained impact in daily life.
  • Incomplete use of physiological data: Due to technical limitations, skin conductance data did not fully capture participants' emotional changes.
  • Future directions: The study could integrate generative AI to add interactive variations for past pianists and further explore extending such interactions to non-musical domains.

Conclusion

This study explores the design of cross-temporal virtual telepresence through MirrorFugue, successfully enhancing users' emotional connections with past pianists and deceased individuals. Participants gained emotional solace, self-reflection, and new insights into lost relationships through the interaction. The research provides significant implications for mental health, technology design, and commemorative interaction, while offering a prototypical approach to expanding the application boundaries of telepresence technology.