Understanding Behind the Smile of Emotion Workers: Detecting After-Call Stress in Call Agents

Paper Title

Understanding Behind the Smile of Emotion Workers: Detecting After-Call Stress in Call Agents

Publication Info

• Topic area: Stress detection in emotion workers using multimodal data and task-aligned modeling.
• Keywords: emotion workers, call agents, stress detection, task-aligned features, multimodal sensing, personalization, machine learning, emotional labor, workplace well-being, ethical AI.

Background and Problem

• Problem / challenge: Existing stress detection systems focus on knowledge workers and use time-fixed sampling, which fails to capture the episodic and interpersonal nature of emotional labor in call agents. There is a lack of research addressing stress detection tailored to the unique workflows of call agents.
• Significance: Call agents face high levels of stress due to emotional labor, which can lead to chronic stress, low job satisfaction, and mental health issues. Effective stress detection is crucial for improving their well-being and workplace conditions.
• Motivation and related work: Prior studies have explored stress detection using physiological and behavioral data but have largely overlooked emotion workers like call agents. Existing approaches often do not align with the structured task cycles of call agents, leaving a gap in understanding stress dynamics in this population.

Solution

• Proposed approach: A task-aligned stress detection framework using multimodal data (task logs, environmental, behavioral, and physiological signals) and personalized machine learning models.
• Novelty:
  1. Introduced task logs as a primary sensing modality for after-call stress detection.
  2. Developed a unified modeling pipeline comparing task-aligned and fixed-time windowing strategies.
  3. Demonstrated the effectiveness of personalized models in capturing individual differences in stress patterns.
  4. Conducted qualitative interviews to contextualize quantitative findings and identify modeling challenges.
• Procedure and key techniques:
  • Conducted a month-long field study with 18 call agents, collecting multimodal data and self-reported stress levels.
  • Compared task-aligned and fixed-time windowing strategies for feature extraction.
  • Evaluated the impact of different data sources (task logs, environmental, behavioral, and self-reports) through ablation studies.
  • Trained machine learning models using nested leave-one-subject-out cross-validation and analyzed feature importance using SHAP values.
  • Conducted semi-structured interviews to gain qualitative insights into stress dynamics and modeling limitations.

Results

• Concrete findings:
  • Task-aligned features performed comparably to 5-minute fixed windows (ROC-AUC: 0.685 vs. 0.692) but outperformed longer windows (e.g., 30 minutes: ROC-AUC 0.605).
  • Task logs were the most informative data source, with features like inquiry length and call duration ranking highest in importance.
  • Personalized models surpassed general models after ∼300–330 personal calls, with ROC-AUC improving from 0.68 to above 0.73 in later folds.
• Advantage over baselines:
  • Task-aligned and short fixed windows captured stress signals more effectively than longer windows.
  • Adding task logs significantly improved model performance (ΔPR-AUC = 0.047).
  • Personalized models revealed heterogeneous feature sets, outperforming general models for most participants.
• Experiments / evaluation:
  • Dataset: 7,442 call-level instances from 15 participants over 310 days.
  • Metrics: ROC-AUC, PR-AUC, weighted F1.
  • Models: Random Forest, XGBoost, CatBoost, and deep learning models like TabNet.
  • Evaluation: Nested leave-one-subject-out cross-validation and statistical significance testing.
• Limitations and future work:
  • Excluded audio and video data due to privacy concerns, limiting the richness of emotional analysis.
  • Findings are specific to the cultural and gendered context of a predominantly female call center in South Korea.
  • Future work should explore larger, multinational datasets and privacy-preserving audio analysis.

Summary

This study developed a task-aligned stress detection framework for call agents, leveraging multimodal data and personalized machine learning models. Task logs emerged as the most critical data source, and task-aligned windowing captured stress signals effectively. Personalized models improved performance after sufficient personal history, highlighting individual variability in stress responses. Qualitative interviews contextualized these findings and revealed challenges beyond sensing, such as unmeasured stressors and behavioral ambiguities. The study advocates for ethical deployment of stress detection tools that prioritize worker well-being while addressing privacy concerns. Future research should expand to diverse contexts and explore additional sensing modalities.