The Influence of Content Modality on Perceptions of Online Misinformation

Research Background and Problem

• Problems or Challenges Identified by the Authors:

  1. Social media has evolved from text-dominated platforms to multimodal environments that include images and videos. However, while rich media (e.g., videos) enhance user engagement, they also significantly increase the spread and credibility of misinformation.
  2. Most existing methods to combat misinformation are text-based and are ineffective against the persuasive power of rich media misinformation.
  3. The authors investigate whether the effectiveness of misinformation correction varies with the modality of the information, particularly whether matching the modality of the misinformation and the correction is necessary.

• Why This Problem is Important: The spread of misinformation harms public understanding and threatens social harmony. As social media users increasingly rely on visual and multisensory content, understanding and optimizing correction strategies to combat misinformation becomes critically urgent.

• Research Motivation and Related Work:

  1. Existing studies suggest that rich media like videos are more persuasive than text, but there is limited research on how modality affects the effectiveness of misinformation correction.
  2. The research is based on Media Richness Theory, exploring the communicative and persuasive power of different modalities (text, images, videos).
  3. Using the social media platform X (formerly Twitter) as a case study, the authors analyze the relationship between correction modality and misinformation modality, an area that remains underexplored in the field of Human-Computer Interaction (HCI).

Solution

• Proposed Solution by the Authors: The authors designed and conducted an experiment involving different modalities to explore the relationship and effectiveness of misinformation and correction modalities. The study specifically examines whether matching the correction modality to the misinformation modality is more effective.

• Innovative Aspects of the Solution:

  1. Proposed the "correction modality matching hypothesis," which posits that the consistency between the correction strategy and the misinformation modality may significantly enhance correction effectiveness.
  2. Developed a 3x3 experimental design involving multiple modalities (text, image, video), offering a more comprehensive approach than most previous studies that focus on one or two modalities.
  3. Systematically examined changes in users' perceived credibility of corrections, incorporating open-ended questions to explore potential factors influencing user judgment.

• Implementation Steps and Key Techniques:

  1. Experimental Design: A factorial experiment with 3 (misinformation modalities: text, image, video) x 3 (correction modalities: text, image, video).
  2. Data Collection:
     • Conducted an online survey using "Prolific," collecting data from 126 participants in the United States.
     • Measured "initial credibility" and "post-correction credibility" through questionnaires and calculated "credibility change."
  3. Analysis Tools:
     • Quantitative analysis employed Generalized Linear Mixed Models (GLMM) to evaluate the effects of correction modalities.
     • Qualitative analysis used Braun and Clarke's thematic analysis to summarize potential factors influencing participants' credibility judgments.

Research Findings

• Specific Findings:

  1. The initial credibility of video-based misinformation was significantly higher than that of text and image-based misinformation.
  2. Significant Modality Matching Effects:
     • Video-based misinformation was most effectively corrected using video corrections.
     • Text-based misinformation was better corrected using text corrections compared to other modalities.
     • For image-based misinformation, there was no significant difference in correction effectiveness across modalities.
  3. Familiarity significantly influenced correction effectiveness: audiences more familiar with the topic of misinformation were more receptive to corrections.

• Comparison with Existing Solutions and Advantages:

  1. This study expands on prior research that focused solely on text-based corrections by comprehensively covering three types of misinformation modalities.
  2. Validated the effects of modality matching, addressing a research gap in the interaction of multimodal misinformation and correction.
  3. Considered participants' individual tendencies (e.g., openness to new ideas, platform usage frequency), making it more complex and nuanced than previous studies.

• Experimental or Evaluation Results:

  • Results support the preliminary accuracy of the "correction modality matching hypothesis."
  • Rich media (videos) not only increase the credibility of misinformation but also require equally rich media forms for the most effective corrections.
  • Qualitative analysis revealed that videos are generally perceived as more understandable and reliable, while text is seen as more direct and simpler to process.

• Limitations and Future Directions:

  1. Limitations:
     • The experiment was limited to the X platform and did not consider cross-platform behavior and preference differences.
     • The image modality was not subdivided (e.g., photographs vs. data visualizations); future research could explore how different types of images affect correction effectiveness.
     • The study did not fully address whether generative AI-era fake videos might reduce audience trust in video content.
  2. Future Directions:
     • Conduct cross-platform studies to verify whether the same modality effects are universally applicable.
     • Introduce more complex and diverse modalities (e.g., audio and interactive media).
     • Explore how other factors, such as duration and content detail, interact with modality to influence credibility shifts.

In summary, this research provides critical insights for designing more effective misinformation intervention strategies on social media platforms, emphasizing the importance of adopting correction strategies that match the modality of the misinformation. This matching approach could significantly enhance user acceptance of the truth.