With Visual Integrity and Care: A Framework for Mixed Methods Research on Visual Social Data

Paper Title

With Visual Integrity and Care: A Framework for Mixed Methods Research on Visual Social Data

Publication Info

• Topic area: Mixed-methods research framework for analyzing visual social data.
• Keywords: Visual social data, mixed methods, visual integrity, computational analysis, human-centered research, visual grammars, problematic information, researcher care, qualitative analysis, HCI.

Background and Problem

• Problem / challenge: Traditional social computing research focuses heavily on textual data, leaving a methodological gap in analyzing visual social data, especially in contexts involving problematic information (e.g., propaganda, hate, misinformation). Existing methods lack systematic frameworks for rigorous, ethical, and scalable visual analysis.
• Significance: Visual content plays a growing role in shaping online and offline social interactions. Addressing this gap is critical for understanding and intervening in the spread of problematic information and for advancing visual research methodologies.
• Motivation and related work: Prior work in visual sociology, media studies, and computational analysis has provided foundational methods but often lacks integration across qualitative and computational approaches. Challenges include decontextualization of data, ethical concerns, and the need for researcher care when analyzing harmful content. This paper builds on these gaps by proposing a comprehensive framework.

Solution

• Proposed approach: A mixed-methods framework for analyzing visual social data, combining visual grammars, human analysis, and computationally supported analysis, with an overarching commitment to researcher care.
• Novelty:
  1. Introduction of a flexible, systematic framework for visual social data analysis.
  2. Emphasis on visual integrity to preserve the visual and social context of data.
  3. Integration of care practices to protect researchers from harm when analyzing distressing content.
  4. Application of the framework across three empirical case studies to demonstrate its adaptability and effectiveness.
• Procedure and key techniques:
  • Visual grammars: Develop systematic structures for analyzing imagery, grounded in research questions and data.
  • Human analysis: Conduct qualitative coding and thematic analysis to generate insights and hypotheses.
  • Computationally supported analysis: Use computational tools to extend human analysis, filter harmful content, and quantify trends while preserving visual integrity.
  • Commitment to care: Implement human and computational infrastructures to minimize researcher harm and optimize analytical labor.

Results

• Concrete findings:
  • Case Study 1: Analyzed 1,500 anti-immigrant propaganda images and 50,000 computationally filtered elements, identifying tactics and trends in real-time.
  • Case Study 2: Investigated 6,000 AI-generated Jesus images, revealing cultural and theological trends through interviews and computational analysis.
  • Case Study 3: Examined 200 Telegram images from the Russia-Ukraine war, developing a multimodal data pipeline to analyze visual solidarity and hostility.
• Advantage over baselines:
  • Enhanced capacity to analyze visual data systematically and ethically.
  • Integration of human insights with computational scalability, preserving visual and social context.
  • Framework supports researcher well-being, reducing exposure to harmful content.
• Experiments / evaluation:
  • Case studies applied the framework across diverse contexts, including propaganda, AI-generated imagery, and wartime visuals.
  • Metrics like inter-rater reliability (e.g., Krippendorf’s Alpha of 0.7041 for solidarity) validated coding consistency.
  • Computational tools extended human analysis, e.g., CLIP for subject detection and modular scripts for trend quantification.
• Limitations and future work:
  • Framework prioritizes transferability over generalizability, limiting its applicability to some research contexts.
  • Focused on Global North contexts; requires adaptation for other cultural settings.
  • Future work includes developing HCI toolkits, addressing video analysis, and exploring broader applications beyond problematic information.

Summary

This paper introduces a mixed-methods framework for analyzing visual social data, integrating visual grammars, human analysis, and computationally supported analysis while prioritizing researcher care. The framework is demonstrated across three case studies involving anti-immigrant propaganda, AI-generated imagery, and wartime visuals, showcasing its adaptability and effectiveness. Key contributions include the concept of visual integrity, pragmatic care practices, and a systematic approach to combining qualitative and computational methods. This framework empowers researchers to study increasingly visual digital spaces while addressing ethical and methodological challenges.