Crystallizing Schemas with Teleoscope: Thematic Curation of Large Text Corpora on Reddit

Paper Title

Crystallizing Schemas with Teleoscope: Thematic Curation of Large Text Corpora on Reddit

Publication Info

• Topic area: Qualitative data curation and analysis for large text corpora.
• Keywords: Thematic curation, qualitative research, large text corpora, interpretivist approach, NLP, semantic search, data visualization, Reddit, thematic schemas, collaborative research.

Background and Problem

• Problem / challenge: Large text corpora, such as those from Reddit, are intractable for qualitative researchers due to their size. Existing statistical subsampling methods for data curation are misaligned with interpretivist qualitative research paradigms.
• Significance: Addressing this gap enables researchers to remain methodologically coherent while working with large datasets, ensuring richer, more nuanced insights into social and cultural phenomena.
• Motivation and related work: Prior systems focus on corpus-level statistical summaries or topic-based overviews, lacking support for iterative, interpretivist curation. This paper builds on advances in AI and NLP to propose a system that aligns data curation with qualitative research principles.

Solution

• Proposed approach: Teleoscope, a web-based interface for thematic curation, enables iterative, interactive, and reflexive refinement of large text corpora into thematic schemas.
• Novelty:
  1. Introduction of thematic curation, an interpretivist sampling strategy resulting in thematic schemas.
  2. Development of an open-source, collaborative data curation platform.
  3. Empirical evaluation of the system through multi-phase deployments with qualitative researchers.
• Procedure and key techniques:
  • Users perform keyword searches to filter the corpus.
  • Documents are grouped, annotated, and iteratively refined using NLP-powered operations (e.g., vector similarity-based ranking, clustering).
  • Outputs include thematic schemas, visual artifacts representing researchers' mental models of document relationships.
  • Collaborative features allow researchers to share, inspect, and modify workflows.

Results

• Concrete findings:
  • Teleoscope supports serendipitous discovery of new keywords and terminologies.
  • Researchers reported increased confidence in achieving search saturation and reduced interpretive bias.
  • Thematic schemas facilitated nuanced discussions and collaborative exploration.
• Advantage over baselines:
  • Enables interpretivist, document-level exploration rather than corpus-level statistical overviews.
  • Provides iterative, human-in-the-loop workflows that align with qualitative research paradigms.
• Experiments / evaluation:
  • Conducted three deployments: a formative study with computer scientists, an extended formative study with qualitative researchers (n=5), and a field deployment with a nursing research team.
  • Evaluations demonstrated usability, methodological coherence, and support for collaborative workflows.
  • Public release included use cases in market research and public policy.
• Limitations and future work:
  • Steep learning curve for qualitative researchers unfamiliar with computational tools.
  • Limited use of general-purpose LLMs; future work could explore their integration.
  • Current focus on Reddit data; future iterations support broader data sources.

Summary

Teleoscope is a web-based system designed to support interpretivist approaches to curating large text corpora, addressing the methodological gap between statistical subsampling and qualitative research principles. By enabling iterative and interactive workflows, it allows researchers to create thematic schemas that reflect their mental models of document relationships. Empirical evaluations demonstrated its effectiveness in fostering serendipitous discovery, achieving search saturation, and supporting collaborative exploration. While challenges such as a steep learning curve remain, Teleoscope shows promise as a scalable, extensible tool for qualitative research across various domains.