Sochiatrist: Signals of Affect in Messaging Data

Messaging is a common mode of communication, with conversations written informally between individuals. Interpreting emotional affect from messaging data can lead to a powerful form of reflection or support clinical therapy. Existing analysis techniques for social media commonly use LIWC and VADER for automated sentiment estimation, which we correlate with PANAS affect scores from 25 participants. These correlations are compared to human review as an upper bound of how well an automated technique could potentially do if it had human judgment. We explore differences in how each technique works, and when they are successful, showing that human review does better than VADER, the best automated technique, when the humans are confident in their labels (0.40 correlation) or they are judging positive affect (0.48 correlation when confident, 0.31 correlation overall). Compared to prior literature, sentiment analysis techniques correlate better with PANAS scores for messaging than social media. These results mean that while any technique that serves as a proxy for PANAS scores has moderate correlation at best, there are some areas to improve the automated techniques by better considering context and timing in conversations.