Drava: Aligning Human Concepts with Machine Learning Latent Dimensions for the Visual Exploration of Small Multiples

Title of the Paper

DRAVA: Aligning Human Concepts with Machine Learning Latent Dimensions for the Visual Exploration of Small Multiples

Paper Information

• Research Domain: Human-AI Collaboration, Explainable Artificial Intelligence (XAI), Data Visualization, Representation Learning
• Keywords: Visual Exploration, XAI, Human-AI Collaboration, Latent Space, Small Multiples

Research Background and Problem

• Identified Problems or Challenges:

  1. Latent vector representations are widely used for data exploration but lack interpretability, making it difficult to align them precisely with human concepts.
  2. While disentangled representation learning (DRL) enhances the interpretability of latent dimensions, the learned latent dimensions may not align with user-understood semantic concepts.
  3. Existing research focuses primarily on model explanation and improvement, neglecting how interpretable latent variables can support concept-driven data exploration.

• Significance:

  1. Latent variables efficiently represent and organize large datasets but cannot be directly used to explain real-world scenarios.
  2. There is an urgent need for data exploration and analysis, especially in high-dimensional domains such as medicine and genomics.

• Motivation and Related Work:

  • Research on disentangled representations (e.g., β-VAE and FactorVAE) has made progress in latent variable interpretability but still faces semantic inconsistency issues.
  • Compared to existing tools, improving the semantic alignment of latent dimensions can help humans intuitively understand data and support complex analytical tasks.

Proposed Solution

• Proposed Method or Framework:

  • Drava is an interactive visual analysis system that enables users to:
    1. Identify discrepancies between latent dimensions and human concepts;
    2. Refine the semantics of latent variables through intuitive interactions;
    3. Perform concept-driven data exploration.
  • A concept adaptor model is used to fine-tune semantic dimensions based on user feedback.

• Innovative Features:

  1. Combines human-centered computing and interactive machine learning, allowing users to directly influence the semantic interpretation of latent variables.
  2. Provides interactive visualization features, including visual clustering, multimodal layouts, and fine-tuning of AI explainability models.
  3. Proposes a clear three-step workflow to help users progressively align latent variables with semantic concepts.

• Implementation Steps and Techniques:

  1. Learning latent representations: DRL is used to extract latent variables and disentangle semantic dimensions.
  2. Users explore latent dimensions through a three-step process:
     • Interpret the semantic dimensions of latent variables;
     • Adjust misaligned data items through drag-and-drop, grouping, or visual previews;
     • Generate new knowledge relevant to the analysis task.
  3. The backend model is trained using β-VAE, while the frontend employs visualization tools like React and Piling.js to create an interactive interface.

Research Outcomes

• Main Achievements:

  1. Provides a ready-to-use tool for analyzing small multiples data, enabling concept-driven data exploration.
  2. Reduces semantic ambiguity by generating synthetic images and grouping latent variables.
  3. Demonstrates the effectiveness of Drava in four use cases, including simple shape data, celebrity datasets, genomic matrices, and breast cancer cell image exploration.

• Advantages Over Existing Solutions:

  • Compared to traditional dimensionality reduction or other tools based on uninterpretable latent variables, Drava is more intuitive and flexible.
  • Allows users to directly fine-tune the semantics of latent variables through interactive operations rather than relying entirely on algorithmic outputs.

• Experimental or Evaluation Results:

  • High accuracy: Demonstrated semantic and interpretability performance of latent dimensions across different dataset scenarios (e.g., skin color and background brightness analysis on the CelebA dataset).
  • Achieved better alignment between user-specified semantic dimensions and actual task requirements.
  • Provided an example of improving breast cancer cell classification performance through user feedback.

• Limitations and Future Directions:

  1. The current method has limited applicability to visually complex concepts and highly diverse datasets (e.g., ImageNet).
  2. Cognitive load increases for users when the number of latent dimensions is large.
  3. Broader user studies are needed to validate usability in real-world scenarios.
  4. Future work will focus on improving visual efficiency (e.g., dynamic sample loading or multi-level detail adjustments).