Simulating Emotions With an Integrated Computational Model of Appraisal and Reinforcement Learning

Title of the Paper

Simulating Emotions With an Integrated Computational Model of Appraisal and Reinforcement Learning

Paper Information

• Subject Area: Affective Computing, Cognitive Modeling, Interactive Learning
• Keywords: Emotion Modeling, Reinforcement Learning, Appraisal Theory, Computational Rationality, User Emotion Prediction, Human-Computer Interaction

Research Background and Problem

• Issues and Challenges:

  1. Traditional emotion prediction methods primarily rely on perceptual data (e.g., physiological signals) but lack understanding of the complex interactions between users' underlying cognitive states and emotional responses.
  2. There is a lack of models capable of dynamically and procedurally simulating emotions; existing models are often static or overly simplified.
  3. The application of integrated models combining Reinforcement Learning (RL) and Appraisal Theory in HCI (Human-Computer Interaction) remains underexplored.

• Significance of the Research:

  1. Emotions are critical factors influencing user behavior and experience in human-computer interaction.
  2. Accurately predicting users' emotional responses can optimize interactive systems, enhancing user experience and engagement.
  3. Developing models that integrate cognition and emotion can provide deeper insights into the mechanisms of emotion formation.

• Motivation and Related Work:

  • This study is inspired by appraisal theory models (e.g., the Component Process Model) and incorporates the reward prediction mechanism of reinforcement learning.
  • Existing literature has explored the relationship between RL and emotional appraisal in static environments but lacks dynamic, interactive validation.
  • The motivation of this work is to design an integrated computational model to simulate the dynamic changes of emotions such as "happiness," "boredom," and "anger" in real interactive scenarios.

Solution

• Methods and Model:

  1. Model Construction:

     • Incorporate appraisal mechanisms from emotion attribution (e.g., goal relevance, novelty, goal conduciveness, and control) into the reinforcement learning framework.
     • Model user interaction as a Markov Decision Process (MDP) and optimize emotion prediction using reinforcement learning algorithms.
     • Use Support Vector Machine (SVM) classification to map computed appraisal value vectors to emotional states such as "happiness," "boredom," and "anger."

  2. Innovations:

     • Link the "temporal difference error (TD error)" in reinforcement learning to quantified emotional appraisals, explaining the elicitation and development of emotions.
     • Propose a dynamic emotion modeling method (introducing a moving average window) to reflect the persistence and multi-stage evolution of emotions.
     • Design tasks and experiments based on human emotion appraisal theories to enhance the model's applicability.

  3. Implementation Steps:

     • Simulate experimental tasks by designing different MDP models to construct scenarios that trigger specific emotions (e.g., "happiness" or "anger").
     • Train RL agents to obtain valuable state-action evaluation data for calculating emotional appraisals.
     • Design experiments involving human participants performing corresponding tasks and collect self-reported emotional data to validate the accuracy of the model's predictions.

Research Outcomes

• Specific Results:

  1. Successfully integrated reinforcement learning with emotional appraisal theory to construct the first comprehensive computational model simulating dynamic emotional changes.
  2. Demonstrated through self-reported data that the model can accurately predict emotions such as "happiness," "boredom," and "anger."

• Advantages Over Existing Solutions:

  1. Dynamic Nature: The model incorporates temporal components, capturing the processual characteristics of emotions rather than merely static snapshots.
  2. Theory-Driven: Based on psychological appraisal theory, the model links cognitive and emotional states in interaction modeling.
  3. Explainability: By using appraisal variables (e.g., goal conduciveness, novelty), the model provides interpretable reasons for emotion predictions.

• Experimental or Evaluation Results:

  • Across multiple experimental tasks, the model's predictions showed high consistency with participants' self-reported data (e.g., R² = 0.78 in Experiment 1, R² = 0.86 in Experiment 2).
  • In challenging or repetitive tasks, the model successfully predicted the temporal dynamics of emotions (e.g., a gradual increase in "boredom" and a decrease in "happiness").

• Limitations and Future Directions:

  1. Limitations:
     • Experimental tasks were simple and did not cover more complex interaction scenarios or a broader range of emotions.
     • The model's dynamic emotion changes were modeled using a simple moving average window, without fully capturing time dependencies.
     • The model has not been extensively adapted to account for individual user differences.
  2. Future Directions:
     • Expand the scope of research to include more types of interactive tasks and emotional scenarios (e.g., anxiety, satisfaction).
     • Enhance the model's temporal dynamics of emotions, such as by introducing time discount factors.
     • Collect more real-world human emotional data in naturalistic settings to validate the model's generalizability and practicality.
     • Explore the integration of physiological signals (e.g., heart rate, skin conductance) to further improve the feasibility of emotion prediction.

Conclusion

• This study provides a novel approach to affective computing by integrating reinforcement learning with emotional appraisal theory, proposing a dynamic and interpretable user emotion prediction model.
• The model holds potential applications in the design of interactive systems, aiding in the understanding of emotion-driven human behavior and enabling emotionally adaptive technological systems.