DoodleTunes: Interactive Visual Analysis of Music-Inspired Children Doodles with Automated Feature Annotation

Title of the Paper

DoodleTunes: Interactive Visual Analysis of Music-Inspired Children Doodles with Automated Feature Annotation

Paper Information

• Research Domain: Interactive Data Visualization and Educational Tool Design
• Keywords: Multimodal Data Visualization, Data Annotation, User Interface, Audio Data, Visual Analytics, Drawing

Research Background and Problem Statement

• Problems or Challenges:
  • The integration of music and visual arts in educational curricula is still in an exploratory phase, with insufficient data-supported theories.
  • Educational experts lack efficient methods to analyze the relationship between music and visual data, as well as the emotional expression and creative processes of children.
  • Automated data annotation is challenging due to the abstract nature of children's doodles and the subjectivity of emotion and style prediction tasks.
• Significance:
  • The integration of music and visual arts in education is crucial for fostering children's artistic creativity and emotional expression. Designing and analyzing such curricula is of great importance for advancing innovation in arts education.
  • Analyzing data to summarize educational theories suitable for children can guide curriculum design and improve the effectiveness of educational products.
• Research Motivation and Related Work:
  • Investigated the current state of music and visual arts integration in education, identifying limitations in implementation and a disconnect between theory and curriculum design.
  • Drew inspiration from deep learning methods and multimodal visualization research in related fields to address the shortcomings of existing approaches.

Proposed Solution

• Proposed Solution:
  • The DoodleTunes system: an interactive visualization system that leverages deep learning methods to analyze children's music-inspired doodle data.
  • The system employs a four-level analytical structure (dataset level, selection level, music level, instance level) to design a step-by-step workflow, supporting incremental data exploration and insight discovery.
• Innovations:
  • Integration of deep learning algorithms for automated annotation of multimodal data features, including doodle semantics, style classification, and emotion prediction.
  • Closely linked visualization modules that enable progressive analysis from overall data to individual data points.
  • Inclusion of data capture and reconstruction of the drawing process, along with a newly added time-series emotion analysis module.
• Implementation Steps and Key Technologies:
  • Designed a Unity-based online doodling software to collect children's doodle data and creative processes.
  • Developed deep learning-based feature annotation methods, such as using SAM and ViT models for semantic annotation and emotion analysis.
  • Built interactive visualization modules, including scatterplot matrices, Sankey diagrams, and time-series emotion change line charts, to intuitively present multimodal data.

Research Outcomes

• Specific Outcomes:
  • Collected 16,543 music-inspired doodles created by students from grades 1 to 3 and completed automated feature annotation using deep learning models.
  • The system supports rapid exploration of potential relationships between children's doodles and music, such as emotional connections, style features, and links to musical spectral characteristics.
  • Provided an advanced analysis workflow design capable of distinguishing between coarse-grained and fine-grained data analysis.
• Comparison with Existing Solutions and Advantages:
  • Compared to simpler music-visual integration methods (e.g., Paint with Music), DoodleTunes emphasizes analytical functionality, offering data-supported theories for arts-integrated education.
  • Automated annotation significantly reduces the cost of manual labeling, and deep learning algorithms outperform traditional CNNs on complex doodle data.
  • The system's emotion time-series analysis and drawing process playback features are unique compared to existing solutions.
• Experimental or Evaluation Results:
  • Feature prediction experiments showed that the accuracy of emotion and style predictions ranged from 70% to 90%, with significant consistency between predictions of doodle and music emotions and human judgments.
  • During system trials, five domain experts highly praised the workflow design and fine-grained process visualization features, while suggesting the addition of manual annotation adjustment functionality.
• Limitations and Future Directions:
  • The system currently supports analysis of only a single dataset at a time; future work could include a dataset comparison feature.
  • The analysis workflow needs to be more flexible, with plans to introduce multi-directional workflows.
  • Enhancing algorithm performance, particularly for highly subjective feature annotation tasks, and enabling manual editing of user features to improve generalizability.