Estimating Shared Mental Models via Communication-Categorized Directed Graphs

Paper Title

Estimating Shared Mental Models via Communication-Categorized Directed Graphs

Publication Info

• Topic area: Continuous estimation of Shared Mental Models (SMM) from team communication data.
• Keywords: Shared Mental Model, Instant Messaging Systems, Graph Neural Networks, Communication Categories, Team Cognition, Large Language Models, Organizational Performance, Continuous Monitoring, Privacy-preserving Annotation, Machine Learning.

Background and Problem

• Problem / challenge: Traditional methods for measuring Shared Mental Models (SMM), such as interviews and questionnaires, are labor-intensive, context-specific, and unsuitable for continuous monitoring. This creates a lack of practical tools for real-time tracking of team cognition.
• Significance: SMM is critical for team performance, enabling coordination and adaptability with minimal communication. Continuous monitoring of SMM could help organizations optimize team dynamics and improve outcomes in various domains.
• Motivation and related work: Prior research has shown that Instant Messaging Systems (IMS) capture essential collaboration signals, but no existing work has identified which communication types correlate with SMM or how SMM can be estimated automatically. This paper addresses these gaps by leveraging IMS data and computational methods.

Solution

• Proposed approach: A novel method to estimate SMM from IMS communication using communication-categorized directed graphs and a graph neural network (GNN).
• Novelty:
  1. A privacy-preserving annotation pipeline using large language models (LLMs) to categorize IMS messages into communicative acts.
  2. Empirical identification of communication categories (Inform, Elicit-inform, Be-positive) associated with SMM.
  3. A GNN-based model that integrates communication categories and directed communication dynamics, outperforming baseline methods.
• Procedure and key techniques:
  • Messages are categorized into communicative acts using LLMs.
  • Category-wise directed communication graphs are constructed, with nodes representing team members and edges weighted by communication dynamics.
  • An extended GNN processes these graphs, incorporating both communication categories and dynamics to estimate SMM.

Results

• Concrete findings:
  • The proposed method achieved an RMSE of 13.50, outperforming baselines (Baseline 1: 47.44, Baseline 2: 21.08, Baseline 3: 16.58, Baseline 4: 21.00).
  • Communication categories (Inform, Elicit-inform, Be-positive) significantly contribute to SMM estimation, while communication dynamics enhance performance when combined with categories.
  • The RMSE corresponds to 13% of the average ground-truth SMM value, indicating room for improvement but demonstrating feasibility for lightweight monitoring.
• Advantage over baselines:
  • Incorporating communication categories improved performance over models that used only conversation content or undifferentiated communication.
  • Combining communication categories and dynamics yielded the best results, highlighting the importance of a holistic approach.
• Experiments / evaluation:
  • Dataset: Slack communication logs and 5-PSMMS questionnaire data from 16 teams in an R&D department.
  • Metrics: Root Mean Square Error (RMSE) for SMM estimation.
  • Baselines compared: Models using only content, undifferentiated communication, or isolated dynamics.
• Limitations and future work:
  • Dataset limited to a single corporate context; broader validation is needed.
  • Annotation quality relies on LLMs, with no manual verification due to privacy constraints.
  • Temporal evolution of communication and team dynamics not explicitly modeled.
  • Applicability to non-corporate domains remains untested.

Summary

This study introduces a novel method for estimating Shared Mental Models (SMM) from Instant Messaging System (IMS) communication using communication-categorized directed graphs and a graph neural network (GNN). The approach leverages large language models (LLMs) for privacy-preserving message categorization and integrates communication categories and dynamics for accurate SMM estimation. Experimental results demonstrate superior performance over baseline methods, with potential applications in corporate, educational, healthcare, and disaster response settings. While promising, the method requires further validation across diverse contexts and improved handling of temporal and dynamic factors.