Lost in Translation: How Does Bilingualism Shape Reader Preferences for Annotated Charts?

Research Background and Problem

• Identified Issues:

  • Data visualization often relies on textual annotations to provide context and guidance, but existing research predominantly focuses on English contexts, neglecting the impact of different languages on information perception and comprehension.
  • Reading complexity in multilingual environments (e.g., cognitive load from language switching) may pose additional challenges to preferences and understanding, which remain underexplored.
  • Current research on best practices for annotation recommendations may not be universally applicable across different languages and cultural contexts.

• Significance:

  • A growing number of global users come from non-English cultural backgrounds, and their multilingual abilities and language preferences may influence their preferences and comprehension of annotated charts.
  • Considering this is crucial for designing inclusive and effective multilingual data visualization tools.

• Research Motivation and Related Work:

  • The authors draw on prior research on the integration of text and charts (e.g., the impact of visual and textual information on user understanding and memory).
  • The motivation of this study is to address the research gap in multilingual user experiences in visualization design and to extend previous work on annotation design to bilingual contexts.

Solution

• Proposed Method or Solution:

  • Design and evaluate a set of experiments to study annotation preferences among bilingual groups with specific language backgrounds (e.g., English + Tamil and English + Arabic).
  • Use six types of charts (bar charts, pie charts, scatter plots, line charts, heatmaps, and maps).
  • Classify annotations by different levels of semantic complexity (L1–L4) and conduct translation and localization.

• Innovations:

  • The first study to investigate the annotation preferences of English-Tamil and English-Arabic bilingual users in data visualizations, incorporating linguistic structures and cultural contexts.
  • Explores the impact of language proficiency (e.g., inner speech, technical thinking, language blending) on annotation preferences and comprehension.
  • Proposes quantitative (preferences and accuracy) and qualitative analyses for annotation design in multilingual environments.

• Implementation Steps and Key Techniques:

  • Use d3.js to create chart stimuli with different annotation levels and semantic content.
  • Translate annotations, ensuring semantic accuracy and cultural adaptability.
  • Experiments include: annotation preference ranking tasks, chart comprehension tests (accuracy of conclusion selection), and semantic complexity analysis of annotations.
  • Use Structural Equation Modeling (SEM) to analyze the statistical impact of annotation preferences and comprehension.

Research Findings

• Specific Findings:

  • Preference Insights:

    • English annotations were more preferred, especially in high-information-density charts (e.g., scatter plots and heatmaps) with high annotation levels.
    • When annotations were provided in the native language, full-text formats were most favored, while complex segmented annotations (e.g., L4) were less preferred.
    • In low-information-density charts (pie charts, bar charts), native language annotations were more appealing.

  • Comprehension Effects:

    • Increasing annotation levels generally improved chart comprehension accuracy, especially in scenarios with full-text annotations in the native language.
    • For complex charts, embedded high-semantic annotations (L3, L4) were more suitable in English, while they imposed cognitive burdens in the native language.

  • Individual Differences Analysis:

    • English proficiency and exposure to English media significantly influenced preferences for English annotations.
    • Users who frequently used inner speech in their native language preferred native language annotations.
    • Language mixing (code-switching) had minimal impact on preferences.

• Advantages of Existing Solutions:

  • Conducted a comprehensive analysis of the complex relationships between language, chart structure, and annotation design, expanding data visualization research from a multilingual perspective for the first time.
  • Provided robust empirical data to support the optimization of data design in multilingual environments.

• Limitations and Future Directions:

  • This study primarily focused on English + Tamil and English + Arabic bilingual environments; future research should expand to other language families to enhance generalizability.
  • Participants were STEM students, which may not represent other age groups or backgrounds.
  • The data visualization topics were relatively neutral; future studies should incorporate more complex emotional or culturally relevant themes.
  • Other design elements (e.g., color, interactivity) and their potential impact on preferences were not deeply explored. Future research should address complex multimodal designs, such as spatial layouts for right-to-left languages.

Conclusion

This study reveals the complex interplay between language, annotation design, and data visualization types. The proposed design recommendations include multilingual switching functionality, hybrid-language annotation design methods, annotation optimization strategies tailored to chart complexity, and providing chart context overviews. These recommendations open new directions for cross-language data visualization design and lay the foundation for future research on inclusive data presentation.