"I Didn't Know I Looked Angry": Characterizing Observed Emotion and Reported Affect at Work

Title of the Paper

"I Didn’t Know I Looked Angry": Characterizing Observed Emotion and Reported Affect at Work

Paper Information

• Research Domain: Affective computing, human-computer interaction, and emotional state studies in workplace settings
• Keywords: emotion, emotion annotation, workplace, automatic emotion recognition, facial expression, expression recognition, context awareness, affective dissonance contrast, user study, psychology

Research Background and Issues

• Problems or Challenges:

  • Automatic Emotion Recognition (AER) technology has been widely applied across various fields, but its accuracy in interpreting emotions remains questionable.
  • The workplace is a critical application scenario for affective computing, yet there is a misalignment between AER performance in this context and subjective emotional self-report data.
  • AER technology often fails to consider internal and external contextual factors (e.g., history, current state, task content) in specific environments, which may impact recognition accuracy.

• Importance of the Research:

  • Accurate emotion recognition technology can enhance workplace productivity and well-being, supporting employees in emotional self-management and team collaboration.
  • Reliable emotional signals are crucial for managing emotional tension, work stress, and task-related scenarios, benefiting both organizations and individuals.

• Motivation and Related Work:

  • AER technology largely relies on basic facial expression classification methods, often developed in controlled laboratory environments (e.g., motion capture or manual annotation), requiring further exploration for practical workplace applications.
  • Previous studies have questioned the universality of facial emotional expressions, particularly in culturally and contextually complex scenarios.

Solution

• Methods or Solutions:

  • This study combines tool usage and diary study methods to analyze the contrast between observed emotional signals from AER technology and self-reported affective signals.
  • Two new metrics are proposed to optimize emotion recognition in workplace settings: emotion spikes and baseline emotions.

• Innovations:

  • Introduces multidimensional emotion analysis, surpassing traditional dominant emotion models of AER systems, by reflecting more nuanced emotional triggers through emotion spikes and baseline emotions.
  • Incorporates contextual variables (e.g., task activities) into observed emotion data, significantly improving the consistency between AER outputs and subjective reports.

• Implementation Steps:

  1. Use AER tools and diaries to record participants' emotional states throughout a workday, with tools continuously capturing facial expressions and workplace context data.
  2. Analyze records and tool outputs from 15 information workers, modeling subjective reports using the Positive and Negative Affect Schedule (PANAS).
  3. Conduct follow-up interviews to compare participants' subjective self-perceptions with AER signals, extracting emotion spikes and baseline emotions from observed data.

Research Findings

• Specific Results:

  • Initial matching accuracy between observed emotional signals and self-reported signals was only 35.4%, but improved to 58.6% after incorporating contextual data and optimizing algorithms.
  • The proposed emotion spikes and baseline emotions metrics outperformed traditional dominant emotion models.
  • Gender differences in emotional expressions were observed, consistent with psychological research—for example, women exhibited more "sadness" baseline emotions, while men showed mixed positive and negative emotions.

• Comparison with Existing Solutions and Advantages:

  • Compared to traditional methods relying solely on dominant emotions, the new approach captures more nuanced patterns of emotional changes.
  • The integration of workplace environmental context into tool-based emotion recognition provides a viable new solution for improvement.

• Experimental or Evaluation Results:

  • Statistical analysis of 215 paired records of observed emotions and self-reported affect validated the significant improvement of the emotion spikes model (relative improvement of 94% compared to dominant emotion models).
  • The report also highlighted the potential applications of dominant emotion and emotion spikes models in workplace scenarios.

• Limitations and Future Directions:

  • The study only examined one day's data, lacking support from long-term longitudinal research.
  • Participants were primarily from research-oriented roles, limiting generalizability to other workplace contexts.
  • AER facial emotion models remain constrained by cultural and contextual biases in their original training data; future work should incorporate more influencing factors and explore personalized emotion labeling strategies.
  • Emphasizes the need for deeper exploration of the relationship between observed and reported signals, alongside ethical considerations in designing emotion technology applications.