Exploratory Visual Analysis of Transcripts for Interaction Analysis in Human-Computer Interaction

Research Background and Issues

• Identified Problems or Challenges:
  • Researchers in Conversation Analysis (CA) and Interaction Analysis (IA) often rely on textual transcripts to analyze participants' multimodal interactions with technology. However, existing methods primarily depend on static, text-based transcripts, which struggle to capture the subtle, dynamic, and multimodal social behaviors in real-time interactions.
  • A significant number of CA and IA studies highlight the lack of dynamic, real-time transcription data visualization tools, especially those capable of reflecting the complexity of interactions (e.g., turn-taking patterns, overlapping speech, social dynamics).
• Significance:
  • Transcript visualization can better assist researchers in understanding the complex social contexts of human-technology interactions and analyzing the relationship between verbal and non-verbal behaviors in conversations. This holds significant value for research, instructional design, and human-computer interaction (HCI) design.
• Research Motivation:
  • Existing qualitative analysis tools (e.g., NVivo, Atlas.ti, and ELAN) lack functionalities for exploring the temporal structure of conversations, participant contributions, and real-time dynamics.
  • While AI-generated transcripts are widely used, their quality and adaptability still require improvement, particularly for refined analyses in CA and IA.
  • The authors aim to develop new visualization methods to facilitate more dynamic and open exploration of transcription data, thereby effectively supporting CA and IA research methodologies.

Solution

• Proposed Solution:
  • Developed an open-source visualization system—Transcript Explorer—which integrates three novel visualization techniques (Distribution Diagrams, Turn Charts, and Contribution Clouds) to dynamically visualize text transcription data linked to video.
  • Explored how these techniques can support interactive viewing, analysis, and exploration of conversations and multimodal interaction behaviors.
• Innovations:
  • Introduced the concept of "Exploratory Visualization of Transcripts Analysis (EVAT)" to provide interactive tools for CA and IA researchers in HCI.
  • The proposed techniques not only support word frequency analysis in conversations but also embed temporal and contextual dimensions, reconnecting text with video to offer researchers a dynamic interactive experience.
• Implementation Steps and Key Techniques:
  1. Four Design Goals:
     • Represent Conversational Parity: Identify when, how long, and to what extent participants contribute.
     • Analyze the Temporal Structure of Speech and Interaction: Study turn-taking, flow, overlaps, and topic development.
     • Track the Evolution of Conceptual Contributions: Observe how concepts are introduced, reused, and stabilized during conversations.
     • Represent Multimodality: Combine video to explore the relationship between non-verbal behaviors (e.g., gestures, attention) and conversations.
  2. Three Technical Applications:
     • Distribution Diagrams: Graphically display the total vocabulary and speaking turns of each participant, supporting turn-taking pattern analysis.
     • Turn Charts: Use ellipses to represent the start and end points of conversations, with height indicating word count, emphasizing sequence and hierarchical structure in turn-taking.
     • Contribution Clouds: Similar to word clouds, integrating temporal information to show word repetition and their contextual contributions to conversation topics.
  3. Interactive Features:
     • Include filtering, animated playback, hover highlighting, and click-to-jump-to-video functionalities, enabling researchers to quickly filter and view key conversation segments.
  4. Implementation Method:
     • The system was developed using JavaScript, compatible with simple text files and more complex structured transcription data (e.g., CSV files with timestamps).

Research Outcomes

• Specific Outcomes:
  1. Developed the open-source Transcript Explorer system, providing multiple conversation visualization views.
  2. Demonstrated the system's flexibility in applications across different domains (e.g., science classroom teaching and presidential debates).
  3. Identified new opportunities for early detection of transcription data deficiencies through visualization tools (e.g., AI transcription sensitivity analysis and quality improvement).
• Comparative Advantages Over Existing Solutions:
  1. Intuitively reflects conversational participation patterns, language flow, and multimodal behaviors, which static text transcripts cannot support.
  2. Provides interactive tools allowing researchers to flexibly switch between macro and micro levels to meet diverse research needs.
  3. Supports real-time video analysis, enabling researchers to dynamically explore non-verbal and physical interaction behaviors.
• Experimental or Evaluation Results:
  1. Focus group interviews revealed high acceptance of the new technologies among participants:
     • Distribution Diagrams highlighted important features such as turn-taking parity and conversation distribution.
     • Turn Charts were described as "visual art," helping researchers uncover the rhythm and participation patterns of conversations.
     • Contribution Clouds supported tracking the evolution of concepts, deepening insights into the development of conversation topics.
  2. Participants suggested further extensions and improvements, including deeper integration of non-verbal data.
• Limitations and Future Directions:
  • The system currently relies heavily on the quality and accuracy of descriptive transcripts, and errors in automated transcription may affect analytical outcomes.
  • While visualization charts streamline the analysis process, they may not fully preserve contextual information for complex phenomena.
  • Future directions include:
    • Further integrating non-verbal behavior data (e.g., gestures and spatial relationships) into the visualization system.
    • Conducting more formal user testing and interdisciplinary expansions.
    • Exploring data privacy and ethics to ensure compliance and legitimacy when dynamically recording interpersonal interactions.