Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

Document Title

Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

Document Information

• Subject Area: Affective computing and human-computer interaction, focusing on improving communication skills in media interviews through nonverbal signals.
• Keywords: Social signals, communication skills training, media interviews, emotion recognition technology, automated feedback

Research Background and Issues

• Identified Problems or Challenges:

  1. Current communication skills training primarily relies on human trainers to provide feedback, which is costly, labor-intensive, and prone to subjectivity.
  2. Most related studies focus on unimodal feedback (e.g., facial expressions) and lack comprehensive analysis of multimodal social signals.
  3. There is insufficient evaluation of skill retention post-training and the long-term impact of automated feedback.

• Importance of the Issues:
  Effective communication is critical for organizational image and success, especially in the context of media interviews. Replacing or enhancing feedback with automated technology can significantly reduce costs and improve fairness and objectivity.

• Research Motivation and Related Work:
  The motivation is to explore how automated technology can improve communication skills training, particularly media interview abilities, by recognizing multimodal nonverbal signals (e.g., facial expressions, tone of voice, gestures). Related work has demonstrated the effectiveness of technology-enhanced communication training in public speaking, job interviews, and classroom teaching, but in-depth research on media interviews remains lacking.

Solution

• Proposed Solution:
  The authors designed a training intervention based on automated recognition of nonverbal signals, integrating multimodal feedback from facial expressions, acoustic signals, gestures, and "trust signals" to enhance communication skills compared to traditional training methods.

• Innovative Aspects:

  1. Introduction of multimodal signal analysis, providing a more comprehensive approach compared to unimodal analysis.
  2. Development of an emotion dashboard and behavior feedback bar chart to explore the utility of "selective feedback," reducing cognitive load.
  3. Comparison of short-term and long-term training effects, including skill retention evaluation six months post-training.

• Implementation Steps and Key Technologies:

  1. Feedback Method: Video playback to facilitate reflection, combined with an automated dashboard providing multimodal signal feedback (e.g., expression scores, voice characteristics).
  2. Tools and Technologies:
     • Facial expressions: Affectiva's AFFDEX system, recognizing emotions through facial action units (Action Units, AU).
     • Acoustic signals: Nemesysco's QA5 system for analyzing voice features.
     • Gestures and trust signals: Captured using Shimmer Unit+ devices and Pentland's sociometric badges to track hand movements and interaction patterns.
  3. Research Design: Pre-test and post-test experiments comparing standard training with technology-enhanced feedback training, evaluated through subjective ratings and quantified social signal changes.

Research Findings

• Specific Results:

  1. Experimental results showed that the social signal feedback group achieved an average 20% improvement in communication skill subjective scores, compared to a 15% improvement in the traditional feedback group.
  2. Social signal feedback significantly reduced participants' negative emotional expressions (e.g., frowning, disgust) during interviews and enhanced engagement (e.g., increased facial activity).
  3. Six months post-training, skill retention experiments indicated that participants in the social signal feedback group were rated as better communicators.

• Advantages Compared to Existing Solutions:

  1. Effective in reducing negative emotional expressions and enhancing engagement during interviews.
  2. Better skill retention post-training compared to traditional methods.
  3. Structured feedback and automated technology reduced subjectivity in human feedback.

• Experimental or Evaluation Results:

  1. Subjective ratings aligned with neutral observer evaluations, with the technology-enhanced feedback group performing better overall.
  2. Multivariate analysis showed significant improvement in participants' performance in media interviews compared to traditional methods.
  3. Data supported that social signal feedback reduced cognitive load and optimized behavioral adjustments.

• Limitations and Future Directions:

  1. Small sample size (22 participants) limits generalizability. Future research should increase sample size, especially incorporating diverse organizational contexts.
  2. Presence of a "ceiling effect," where participants with excellent baseline skills showed limited improvement. Future studies could exclude high-scoring baseline participants to control for this effect.
  3. Further exploration of long-term impacts is needed, such as extending follow-up periods and investigating behavioral threshold effects of different feedback methods.

This study opens new perspectives for the application of automated technology in professional skills training, demonstrating the potential for deep integration of technology and education.