Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples

Document Title

Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples

Document Information

• Subject Area: Data Visualization and Machine Learning Embedding Model Comparison
• Keywords: Embedding Space, Data Visualization, Interactive Systems, Small Multiples, Neighborhood Computation, Similarity Comparison

Research Background and Problem

• Identified Problems:

  • Comparing embedding models is a critical task in machine learning deployment or downstream analysis, but existing methods are often cumbersome and fail to systematically reveal the characteristics of embedding spaces.
  • Current techniques primarily focus on single-model analysis or global embedding structure comparison, lacking methods to simultaneously explore global and local substructures.
  • Users often cannot quickly generate hypotheses and verify results within embedding spaces. Existing tools lack sufficient interactivity, relying on manual object specification, leading to lengthy and error-prone processes.

• Significance:

  • Embedding models are used in fields such as Natural Language Processing (NLP), computational biology, and recommendation systems. Comparative analysis of these models can reveal semantic changes, the impact of training data and architectures, and directions for model optimization.
  • A comprehensive understanding of embedding spaces is crucial for evaluating model performance and guiding domain experts in identifying models that capture target semantics for downstream tasks.

• Motivation and Related Work:

  • Current dimensionality reduction-based methods (e.g., PCA, t-SNE, UMAP) and neighborhood analysis approaches fail to provide systematic comparisons.
  • Existing literature on embedding space comparison methods, such as direct alignment algorithms and task-specific approaches, are not flexible enough and are difficult to generalize across diverse domains.

Solution

• Method and Innovation:

  • Propose the Embedding Comparator, an interactive system that combines global embedding space visualization with local neighborhood comparison to support systematic analysis between embedding models.
  • Introduce a Local Neighborhood Similarity (LNS) metric, which quantifies the similarity of embedding objects between two models by calculating the intersection of their k-nearest neighbors.
  • Use small multiples (local neighborhood dominoes) to display the local substructures of embedding objects, quickly revealing model similarities and differences.

• Implementation Steps:

  • Implement global embedding space projections (PCA, t-SNE, UMAP) to display the geometric structure of embedding spaces.
  • Precompute neighborhood similarity for all embedding objects and encode this information through histograms and embedding scatterplots, showing distributions and the most/least similar objects.
  • Design interactive features allowing users to filter objects, select local neighborhoods, view small multiples, and link global and local views to support rapid iterative analysis.

Research Outcomes

• Specific Outcomes:

  • Embedding Comparator was validated through case studies and user experiments, including:
    • Revealing semantic changes induced by fine-tuning in sentiment analysis tasks (e.g., shifts in the emotional meaning of numbers).
    • Discovering historical semantic shifts in words like "gay" and "aids" from 1800 to 2000 in language evolution studies.
    • Demonstrating how language and vision embedding models capture semantic and appearance similarities, respectively, in multimodal tasks.

• Advantages:

  • Enables systematic comparison of embedding models without requiring task-specific metrics or model alignment.
  • Improves existing tool workflows, significantly reducing time and effort compared to manual methods.
  • Allows both data-driven and model-driven users to quickly generate insights and hypotheses.

• Experimental or Evaluation Results:

  • User experiments showed that researchers using the Embedding Comparator generated significantly more insights (average 10.7) compared to traditional tools (average 4.1), with insight generation time reduced from 8 minutes to under 1 minute.
  • Users were able to quickly generate theories and successfully validate their hypotheses.

• Limitations and Future Directions:

  • The current system struggles with handling long object labels (e.g., chemical molecule SMILES). Future work could explore more compact object representations.
  • Potential to expand to more embedding comparison scenarios, such as multi-faceted comparisons of n models or incorporating training data context for a more complete analysis.
  • Optimize ranking algorithms to prioritize objects with significant differences across more models.