Templates and Trust-o-meters: Towards a widely deployable indicator of trust in Wikipedia

Title of the Paper

Templates and Trust-o-meters: Towards a widely deployable indicator of trust in Wikipedia

Bibliographic Information

• Authors: Andrew Kuznetsov, Margeigh Novotny, Jessica Klein, Diego Saez-Trumper, Aniket Kittur
• Publication Year: 2022
• Conference Name: CHI Conference on Human Factors in Computing Systems (CHI ’22)
• Location: New Orleans, LA, USA
• DOI: 10.1145/3491102.3517523
• Research Domain: User Generated Content (UGC), Information Trust, Wikipedia Content Management
• Keywords: Wikipedia, trust indicators, user interface design, automated quality assessment, content reliability, user behavior, human-computer interaction interface, online content quality, information transparency

Research Background and Problem Statement

• Challenges and Problems: User-generated content (e.g., Wikipedia) faces trust issues due to the openness of the community. Although numerous trust indicator studies exist, these approaches often encounter practical deployment challenges, such as designing trust markers that are both prominent and unobtrusive within limited screen space, and calibrating trust levels corresponding to varying content quality.
• Importance: Establishing trust indicators can reduce user reliance on low-quality content while increasing confidence in high-quality content, thereby enhancing user experience and fostering long-term community growth.
• Research Motivation:
  • Existing indicators are often limited to laboratory settings and fail to address the “last-mile” problem in real-world deployment.
  • Current templates tend to focus on warning users about issues, lacking mechanisms to significantly enhance trust in high-quality content.
  • Designing mechanisms to globally improve readers’ trust in Wikipedia (and similar platforms) with broad deployment potential.

Proposed Solution

• Solution Overview:

  • Experimentally validate the impact of existing Wikipedia templates and newly introduced trust indicators on user trust.
  • Design a novel “Trust-o-meter” to convey trust information in a compact, prominent, yet minimally intrusive manner.
  • Integrate multiple quality signals, combining positive and negative content indicators into a simple, readable metric.

• Innovations:

  • Development of a “composite trust measurement tool” suitable for both experimental and practical applications.
  • Experimental scope includes various signals (e.g., edit disputes, citation credibility, author activity) and their effects across different trust levels.
  • Identification of user trust response curves and behavioral characteristics.

• Implementation Steps:

  1. Experiment 1: Analyze the impact of individual problem templates and new indicators on user trust in content.
  2. Experiment 2: Explore the visual prominence and user engagement of different Trust-o-meter design options.
  3. Experiment 3: Design and validate the composite Trust-o-meter, testing its effects on both content trust and overall trust in Wikipedia.
  4. Analyze behavioral patterns and trust models of surveyed users during implementation to understand the adaptability of indicators to diverse user backgrounds and perceptions.

Research Findings

• Specific Findings:

  • Commonly used Wikipedia templates (e.g., marking advertising content or conflicts of interest) significantly reduce user trust; newly designed “disputed citation” indicators also effectively lower trust.
  • Positive signals (e.g., high-quality ratings, non-disputed content) did not significantly enhance trust in individual articles, but the composite Trust-o-meter successfully addressed this issue.
  • Experiments show that readers unfamiliar with Wikipedia’s content generation process react more extremely to trust indicators, whereas familiar users exhibit more balanced responses.

• Comparison with Existing Solutions:

  • Current templates primarily aim to warn readers, while the tools designed in this study emphasize trust enhancement.
  • The authors clarified the feasibility of transitioning trust indicators from experimental settings to real-world deployment.

• Experimental and Evaluation Results:

  • Negative signals such as “disputed citation” significantly reduced trust (maximum decrease of 1.601 points), while positive signals alone did not effectively increase trust; however, composite trust indicators for high-quality articles (e.g., “green level”) improved trust by approximately +0.95 points.
  • Reader attention was strongly correlated with the placement, size, and animation effects of indicators (“top-right corner blinking” method was the most prominent, attracting attention up to 55%).
  • Trust reduction was more pronounced in disputed articles, highlighting the need for tiered designs tailored to different reader scenarios.

• Limitations and Future Directions:

  • Limitations: Current experiments used long and content-rich sample articles, leaving a research gap regarding trust indicators for short articles and low-traffic content.
  • Future Directions:
    • Expand the adaptability of trust indicators to global content communities, such as open-source software projects.
    • Explore community-driven trust scoring systems, incorporating editors into the loop (community-in-the-loop design).
    • Combine natural language processing techniques to provide more efficient implementation paths for signals that are difficult to quantify (e.g., reference disputes).