Script-Strategy Aligned Generation: Aligning LLMs with Expert-Crafted Dialogue Scripts and Therapeutic Strategies for PsychotherapyChatbots or conversational agents (CAs) are increasingly used to improve access to digital psychotherapy. Many current systems rely on rigid, rule-based designs, heavily dependent on expert-crafted dialogue scripts for guiding therapeutic conversations. Although advances in large language models (LLMs) offer potential for more flexible interactions, their lack of controllability and explanability poses challenges in psychotherapy. In this work, we explored how aligning LLMs with expert-crafted scripts can enhance psychotherapeutic chatbot performance. Our comparative Study 1 showed that LLMs aligned with expert-crafted scripts through prompting and fine-tuning significantly outperformed both pure LLMs and rule-based chatbots, achieving an effective balance between dialogue flexibility and adherence to therapeutic principles. Building on findings, we proposed ``Script-Strategy Aligned Generation (SSAG)'', a more flexible alignment approach that reduces reliance on fully scripted content while maintaining LLMs' therapeutic adherence and controllability. In a 10-day field Study 2, SSAG demonstrated performance comparable to full script alignment, empirically supporting SSAG as an efficient approach for aligning LLMs with domain expertise. Our work advances LLM applications in psychotherapy by providing a controllable and scalable solution, reducing reliance on expert effort. It also provides a collaborative framework for domain experts and developers to efficiently build expertise-aligned chatbots, broadening access to broader context of psychotherapy.2025XSXin Sun et al.Facilitating Support and BelongingCSCW
Haptic Biosignals Affect Proxemics Toward Virtual Reality AgentsEncounters with virtual agents currently lack the haptic viscerality of human contact. While digital biosignal communication can mediate such virtual social interactions, how artificial haptic biosignals influence users’ personal space during Virtual Reality (VR) experiences is unknown. Designing vibrotactile heartbeats and thermally-actuated body temperature, we ran a within-subjects study (N=31) to investigate feedback (Thermal, Vibration, Thermal+Vibration, None) and agent stories (Negative, Neutral, Positive) on objective and subjective interpersonal distance (IPD), perceived arousal and comfort, presence, and post-experience responses. Findings showed that thermal feedback decreased objective but not subjective IPD, whereas vibrotactile heartbeats (signaling agent's closeness) increased both while heightening arousal and discomfort. Agents' stories did not affect IPD, arousal, or comfort. Our qualitative findings shed light on signal ambiguity and presence constructs within VR-based haptic stimulation. We contribute insights into artificial biosignals and their influence on VR proxemics, with cautionary considerations should the boundaries blur between physical and virtual touch.2025SOSimone Ooms et al.Utrecht University, Human-Centred Computing; Centrum Wiskunde & Informatica, Distributed & Interactive SystemsVibrotactile Feedback & Skin StimulationSocial & Collaborative VRImmersion & Presence ResearchCHI
User Experience Design Professionals’ Perceptions of Generative Artificial IntelligenceAmong creative professionals, Generative Artificial Intelligence (GenAI) has sparked excitement over its capabilities and fear over unanticipated consequences. How does GenAI impact User Experience Design (UXD) practice, and are fears warranted? We interviewed 20 UX Designers, with diverse experience and across companies (startups to large enterprises). We probed them to characterize their practices, and sample their attitudes, concerns, and expectations. We found that experienced designers are confident in their originality, creativity, and empathic skills, and find GenAI’s role as assistive. They emphasized the unique human factors of “enjoyment” and “agency”, where humans remain the arbiters of “AI alignment”. However, skill degradation, job replacement, and creativity exhaustion can adversely impact junior designers. We discuss implications for human-GenAI collaboration, specifically copyright and ownership, human creativity and agency, and AI literacy and access. Through the lens of responsible and participatory AI, we contribute a deeper understanding of GenAI fears and opportunities for UXD.2024JLJie Li et al.EPAMGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
ShareYourReality: Investigating Haptic Feedback and Agency in Virtual Avatar Co-embodimentVirtual co-embodiment enables two users to share a single avatar in Virtual Reality (VR). During such experiences, the illusion of shared motion control can break during joint-action activities, highlighting the need for position-aware feedback mechanisms. Drawing on the perceptual crossing paradigm, we explore how haptics can enable non-verbal coordination between co-embodied participants. In a within-subjects study (20 participant pairs), we examined the effects of vibrotactile haptic feedback (None, Present) and avatar control distribution (25-75%, 50-50%, 75-25%) across two VR reaching tasks (Targeted, Free-choice) on participants’ Sense of Agency (SoA), co-presence, body ownership, and motion synchrony. We found (a) lower SoA in the free-choice with haptics than without, (b) higher SoA during the shared targeted task, (c) co-presence and body ownership were significantly higher in the free-choice task, (d) players’ hand motions synchronized more in the targeted task. We provide cautionary considerations when including haptic feedback mechanisms for avatar co-embodiment experiences.2024KVKarthikeya Puttur Venkatraj et al.Centrum Wiskunde & Informatica, Delft University of TechnologyMid-Air Haptics (Ultrasonic)Social & Collaborative VRIdentity & Avatars in XRCHI
Affective Driver-Pedestrian Interaction: Exploring Driver Affective Responses Toward Pedestrian Crossing Actions Using Camera and Physiological SensorsEliciting and capturing drivers' affective responses in a realistic outdoor setting with pedestrians poses a challenge when designing in-vehicle, empathic interfaces. To address this, we designed a controlled, outdoor car driving circuit where drivers (N=27) drove and encountered pedestrian confederates who performed non-verbal positive or non-positive road crossing actions towards them. Our findings reveal that drivers reported higher valence upon observing positive, non-verbal crossing actions, and higher arousal upon observing non-positive crossing actions. Drivers' heart signals (BVP, IBI and BPM), skin conductance and facial expressions (brow lowering, eyelid tightening, nose wrinkling, and lip stretching) all varied significantly when observing positive and non-positive actions. Our car driving study, by drawing on realistic driving conditions, further contributes to the development of in-vehicle empathic interfaces that leverage behavioural and physiological sensing. Through automatic inference of driver affect resulting from pedestrian actions, our work can enable novel empathic interfaces for supporting driver emotion self-regulation.2023SRShruti Rao et al.In-Vehicle Haptic, Audio & Multimodal FeedbackHuman Pose & Activity RecognitionAutoUI
Understanding and Designing Avatar Biosignal Visualizations for Social Virtual Reality EntertainmentVisualizing biosignals can be important for social Virtual Reality (VR), where avatar non-verbal cues are missing. While several biosignal representations exist, designing effective visualizations and understanding user perceptions within social VR entertainment remains unclear. We adopt a mixed-methods approach to design biosignals for social VR entertainment. Using survey (N=54), context-mapping (N=6), and co-design (N=6) methods, we derive four visualizations. We then ran a within-subjects study (N=32) in a virtual jazz-bar to investigate how heart rate (HR) and breathing rate (BR) visualizations, and signal rate, influence perceived avatar arousal, user distraction, and preferences. Findings show that skeuomorphic visualizations for both biosignals allow differentiable arousal inference; skeuomorphic and particles were least distracting for HR, whereas all were similarly distracting for BR; biosignal perceptions often depend on avatar relations, entertainment type, and emotion inference of avatars versus spaces. We contribute HR and BR visualizations, and considerations for designing social VR entertainment biosignal visualizations.2022SLSueyoon Lee et al.Centrum Wiskunde & Informatica (CWI), Delft University of TechnologySocial & Collaborative VRImmersion & Presence ResearchIdentity & Avatars in XRCHI
RCEA-360VR: Real-time, Continuous Emotion Annotation in 360° VR Videos for Collecting Precise Viewport-dependent Ground Truth LabelsPrecise emotion ground truth labels for 360° virtual reality (VR) video watching are essential for fine-grained predictions under varying viewing behavior. However, current annotation techniques either rely on post-stimulus discrete self-reports, or real-time, continuous emotion annotations (RCEA) but only for desktop/mobile settings. We present RCEA for 360° VR videos (RCEA-360VR), where we evaluate in a controlled study (N=32) the usability of two peripheral visualization techniques: HaloLight and DotSize. We furthermore develop a method that considers head movements when fusing labels. Using physiological, behavioral, and subjective measures, we show that (1) both techniques do not increase users' workload, sickness, nor break presence (2) our continuous valence and arousal annotations are consistent with discrete within-VR and original stimuli ratings (3) users exhibit high similarity in viewing behavior, where fused ratings perfectly align with intended labels. Our work contributes usable and effective techniques for collecting fine-grained viewport-dependent emotion labels in 360° VR.2021TXTong Xue et al.Beijing Institute of Technology, Centrum Wiskunde & Informatica (CWI)Eye Tracking & Gaze InteractionSocial & Collaborative VRImmersion & Presence ResearchCHI
RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth LabelsCollecting accurate and precise emotion ground truth labels for mobile video watching is essential for ensuring meaningful predictions. However, video-based emotion annotation techniques either rely on post-stimulus discrete self-reports, or allow real-time, continuous emotion annotations (RCEA) only for desktop settings. Following a user-centric approach, we designed an RCEA technique for mobile video watching, and validated its usability and reliability in a controlled, indoor (N=12) and later outdoor (N=20) study. Drawing on physiological measures, interaction logs, and subjective workload reports, we show that (1) RCEA is perceived to be usable for annotating emotions while mobile video watching, without increasing users' mental workload (2) the resulting time-variant annotations are comparable with intended emotion attributes of the video stimuli (classification error for valence: 8.3%; arousal: 25%). We contribute a validated annotation technique and associated annotation fusion method, that is suitable for collecting fine-grained emotion annotations while users watch mobile videos.2020TZTianyi Zhang et al.Centrum Wiskunde & Informatica and Delft University of TechnologyVisualization Perception & CognitionBiosensors & Physiological MonitoringCHI
ThermalWear: Exploring Wearable On-chest Thermal Displays to Augment Voice Messages with AffectVoice is a rich modality for conveying emotions, however emotional prosody production can be situationally or medically impaired. Since thermal displays have been shown to evoke emotions, we explore how thermal stimulation can augment perception of neutrally-spoken voice messages with affect. We designed ThermalWear, a wearable on-chest thermal display, then tested in a controlled study (N=12) the effects of fabric, thermal intensity, and direction of change. Thereafter, we synthesized 12 neutrally-spoken voice messages, validated (N=7) them, then tested (N=12) if thermal stimuli can augment their perception with affect. We found warm and cool stimuli (a) can be perceived on the chest, and quickly without fabric (4.7-5s) (b) do not incur discomfort (c) generally increase arousal of voice messages and (d) increase / decrease message valence, respectively. We discuss how thermal displays can augment voice perception, which can enhance voice assistants and support individuals with emotional prosody impairments.2020AAAbdallah El Ali et al.Centrum Wiskunde & InformaticaHaptic WearablesCHI
NaviBike: Comparing Unimodal Navigation Cues for Child CyclistsNavigation systems for cyclists are commonly screen-based devices mounted on the handlebar which show map information. Typically, adult cyclists have to explicitly look down for directions. This can be distracting and challenging for children, given their developmental differences in motor and perceptual-motor abilities compared with adults. To address this issue, we designed different unimodal cues and explored their suitability for child cyclists through two experiments. In the first experiment, we developed an indoor bicycle simulator and compared auditory, light, and vibrotactile navigation cues. In the second experiment, we investigated these navigation cues in-situ in an outdoor practice test track using a mid-size tricycle. To simulate road distractions, children were given an additional auditory task in both experiments. We found that auditory navigational cues were the most understandable and the least prone to navigation errors. However, light and vibrotactile cues might be useful for educating younger child cyclists.2019AMAndrii Matviienko et al.OFFIS - Institute for Information TechnologyIn-Vehicle Haptic, Audio & Multimodal FeedbackMicromobility (E-bike, E-scooter) InteractionCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)CHI
Measuring and Understanding Photo Sharing Experiences in Social Virtual RealityMillions of photos are shared online daily, but the richness of interaction compared with face-to-face (F2F) sharing is still missing. While this may change with social Virtual Reality (socialVR), we still lack tools to measure such immersive and interactive experiences. In this paper, we investigate photo sharing experiences in immersive environments, focusing on socialVR. Running context mapping (N=10), an expert creative session (N=6), and an online experience clustering questionnaire (N=20), we develop and statistically evaluate a questionnaire to measure photo sharing experiences. We then ran a controlled, within-subject study (N=26 pairs) to compare photo sharing under F2F, Skype, and Facebook Spaces. Using interviews, audio analysis, and our questionnaire, we found that socialVR can closely approximate F2F sharing. We contribute empirical findings on the immersiveness differences between digital communication media, and propose a socialVR questionnaire that can in the future generalize beyond photo sharing.2019JLJie Li et al.Centrum Wiskunde & InformaticaSocial & Collaborative VRImmersion & Presence ResearchInteractive Narrative & Immersive StorytellingCHI
Measuring, Understanding, and Classifying News Media Sympathy on Twitter after Crisis EventsThis paper investigates bias in coverage between Western and Arab media on Twitter after the November 2015 Beirut and Paris terror attacks. Using two Twitter datasets covering each attack, we investigate how Western and Arab media differed in coverage bias, sympathy bias, and resulting information propagation. We crowdsourced sympathy and sentiment labels for 2,390 tweets across four languages (English, Arabic, French, German), built a regression model to characterize sympathy, and thereafter trained a deep convolutional neural network to predict sympathy. Key findings show: (a) both events were disproportionately covered (b) Western media exhibited less sympathy, where each media coverage was more sympathetic towards the country affected in their respective region (c) Sympathy predictions supported ground truth analysis that Western media was less sympathetic than Arab media (d) Sympathetic tweets do not spread any further. We discuss our results in light of global news flow, Twitter affordances, and public perception impact.2018AAAbdallah El Ali et al.CWI (Centrum Wiskunde & Informatica), University of OldenburgSocial Platform Design & User BehaviorContent Moderation & Platform GovernanceMisinformation & Fact-CheckingCHI