Metrics for Peer Counseling: Triangulating Success Outcomes for Online Therapy Platforms

Title of the Paper

Metrics for Peer Counseling: Triangulating Success Outcomes for Online Therapy Platforms

Paper Information

• Subject Area: Evaluating the effectiveness of peer counseling on online mental health platforms
• Keywords: Online mental health, peer counseling, linguistic predictors, outcome measurement, data analysis, topic matching, interaction design, mental health assessment, conversational outcomes, OMHP
• Authors and Affiliations:
  • Tony Wang (Georgia Institute of Technology)
  • Haard K. Shah (7 Cups of Tea)
  • Raj Sanjay Shah (Georgia Institute of Technology)
  • Yi-Chia Wang (Independent Researcher)
  • Robert E. Kraut (Carnegie Mellon University)
  • Diyi Yang (Stanford University)

Research Background and Problem

• Key Issues or Challenges: The effectiveness of peer counseling on online mental health platforms (OMHPs) lacks standardized metrics. Previous studies often rely on single indicators to define success, which may lead to inconsistent results and limited generalizability.
• Significance:
  • OMHPs provide low-cost and convenient mental health support, playing a crucial role in reducing societal stigma surrounding mental health.
  • For platform designers, understanding how to measure successful conversations can help improve user experience and enhance service quality.
• Research Motivation: To address the limitations of single measurement methods in past studies, the authors propose evaluating the effectiveness of peer counseling through multiple metrics to gain deeper academic and practical insights.
• Related Work: Relevant literature in behavioral economics increasingly emphasizes triangulating success outcomes through multiple diagnostic signals to improve the effectiveness of mental health prediction models. However, research on OMHPs still lacks systematic comparisons using multi-metric approaches.

Solution

• Methods and Solutions:
  • Systematically integrate success outcome metrics at three levels—community-level, conversation-level, and individual-level—and propose the "Outcome Triangulation" method.
  • Utilize a large dataset containing 1.73 million conversations, combining statistical modeling (Logistic regression, Heckman selection model, Ordinal regression, etc.) to analyze key linguistic predictors and their relationships with four success metrics.
• Innovations:
  • The first systematic triangulation of success outcomes in OMHP research.
  • Proposed four core metrics: community retention, conversation follow-up rate, counselor ratings, and mood changes, to deepen understanding of platform-level decision-making and evaluation.
• Implementation Steps:
  1. Data Collection and Cleaning: Analyze private chat data from the 7 Cups platform between 2020 and 2022.
  2. Variable Definition and Operationalization: Define and operationalize behavioral and attitudinal metrics.
  3. Model Testing and Regression Analysis: Use various statistical methods to systematically analyze predictors of the four success metrics.
  4. Result Analysis and Comparison: Compare the consistency and potential tensions among key predictors across different metrics.

Research Findings

• Specific Findings:
  1. Low Correlation Among Metrics:
     • The correlation among the four success metrics is low (e.g., retention rate and follow-up rate τ=-0.03), suggesting that different metrics capture distinct definitions of success.
  2. Predictive Consistency Variations Across Metrics:
     • Predictors of conversational outcomes (e.g., follow-up, ratings) such as total word count and topic matching exhibit strong internal consistency but differ from predictors of community retention metrics.
     • Member self-disclosure and linguistic style matching have varying directional impacts on follow-up and rating outcomes.
  3. Limited Association with Mood Changes: The effect of conversations on improving members' mood is not significant, consistent with findings in previous literature.
• Comparison with Existing Solutions:
  • This study is the first to systematically reveal the relationships and trade-offs between different definitions and methods of success outcomes in peer counseling research.
  • Provides a new methodological framework to complement existing studies that focus on single-outcome evaluations of peer counseling.
• Experiment and Evaluation Results:
  • Indicates that "single-session therapy" (SST) may align naturally with the service model of OMHPs.
• Limitations and Future Directions:
  • This study does not directly investigate causal relationships; future research could explore the true causal effects of influencing factors.
  • Since the experimental platform used was 7 Cups, future studies should compare results across different types of platforms.
  • Further expansion to include instant feedback metrics (e.g., likes/support features on social media) as novel measures for peer counseling outcomes.

Through this study, the authors emphasize that a systematic outcome triangulation method is a necessary supplement for understanding the success of peer counseling on OMHPs, offering new opportunities to improve platform user interaction and support experiences.