The Evolving Duet of Two Modalities: A Survey on Integrating Text and Visualization for Data Communication

Paper Title

The Evolving Duet of Two Modalities: A Survey on Integrating Text and Visualization for Data Communication

Publication Info

• Topic area: Integration of text and visualization in data communication.
• Keywords: Text as narrative, data visualization, storytelling, text design, text-visualization integration, empirical studies, narrative tasks, interactive systems, large language models, data communication.

Background and Problem

• Problem / challenge: Research on the role of text in data visualization is fragmented, often conflating its roles as data input, interaction modality, and narrative device. The specific use of text as a narrative medium remains underexplored and unsynthesized.
• Significance: Text is essential for enhancing comprehension, engagement, and memorability in data visualization. Understanding its role is critical for designing effective communication tools across domains like journalism, education, and research.
• Motivation and related work: Prior surveys have extensively reviewed text as data and interaction but lack a systematic investigation into text as a narrative device. Existing studies on data storytelling and visualization design often overlook the communicative function of textual components, leaving a gap in actionable guidance for practitioners.

Solution

• Proposed approach: A systematic review of 98 academic papers to investigate the integration of text as a narrative device in data visualization.
• Novelty:
  1. Consolidates fragmented knowledge into a three-stage framework (why, what, how) for text design in data visualization.
  2. Identifies eight textual forms and their integration modalities with visualization.
  3. Proposes five primary narrative tasks (explain, emphasize, couple, adapt, verify) and catalogs design techniques for each.
  4. Highlights research gaps and opportunities for future work, particularly in emerging areas like AI-assisted narratives and dynamic text-visual relationships.
• Procedure and key techniques:
  • Curated a corpus of 98 papers from major visualization and HCI venues using targeted keyword searches.
  • Analyzed the papers along three dimensions: motivations (why), manifestations (what), and design techniques (how).
  • Developed a taxonomy of textual forms, integration modalities, and narrative tasks.

Results

• Concrete findings:
  • Identified eight textual forms (e.g., titles, labels, annotations, descriptions) and three integration modalities (text in visualization, hand-in-hand, visualization in text).
  • Categorized design techniques into five narrative tasks:
    • Explain: Organizing text with narrative structures, adding context, managing perspectives, and enriching communication styles.
    • Emphasize: Highlighting key entities, enhancing readability, and augmenting semantics.
    • Couple: Inline enhancements, text-visualization references, and mutual translation.
    • Adapt: Supporting multiple integration formats, format transfer, and scenario-specific designs.
    • Verify: Manual critique and refinement, verification cues.
  • Empirical studies show text improves comprehension, memory, and engagement but highlight inconsistencies in metrics like efficiency and attitude change.
• Advantage over baselines:
  • Provides a structured framework for understanding and designing text in data visualization, addressing gaps in prior fragmented research.
  • Highlights underexplored areas like bidirectional text-visual coupling, adaptive formats, and verification in AI-generated narratives.
• Experiments / evaluation:
  • Synthesized empirical findings from 23 studies, covering metrics like comprehension, memory, trust, and gaze behavior.
  • Demonstrated the effectiveness of text-visual integration through controlled comparisons and user studies.
• Limitations and future work:
  • Limited to papers from specific venues; may have missed relevant work.
  • Rapid advancements in AI and LLMs may necessitate updates to the taxonomy.
  • Calls for deeper empirical studies, exploration of emerging mediums (e.g., VR, physicalization), and techniques for verifying AI-generated narratives.

Summary

This survey systematically investigates the role of text as a narrative medium in data visualization, addressing gaps in prior research. It categorizes textual forms, integration modalities, and design techniques into a comprehensive framework of five narrative tasks: explain, emphasize, couple, adapt, and verify. Empirical findings highlight text’s value in enhancing comprehension, engagement, and trust, while identifying inconsistencies and underexplored areas like dynamic text-visual relationships and AI-assisted narratives. The work provides actionable guidance for researchers and practitioners, with significant implications for designing effective and trustworthy data communication tools.