Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization

Research Background and Problem

• Identified Problems or Challenges by the Authors: Determining the similarity between different visualizations is a critical issue in many applications, such as visualization search, recommendation systems, and automated design and ranking. However, human perception of visualization similarity often involves complex cognitive processes that are difficult to fully capture using traditional pixel-level and structured techniques. Additionally, collecting human judgment data is costly and limited to specific experimental designs.
• Significance: Similarity judgments directly impact the effectiveness of information design and communication, enhancing the efficiency of visualization tools. This can make visualization design more directly reflective of the true relationships within the data.
• Research Motivation: Previous studies have shown that similarity metrics based on deep features perform exceptionally well in image tasks, better capturing human perceptions of image similarity. Therefore, the authors attempt to apply these techniques to the field of information visualization, exploring whether such metrics can effectively handle similarity judgments for scatterplots and other basic visual elements.

Solution

• Proposed Method:
  • Extend existing deep feature similarity metrics using five deep learning (DL) architectures and three pre-trained weights (including ImageNet and Stylized ImageNet).
  • Compare with the gradient descent-based MS-SSIM method and replicate two data-driven experiments aimed at studying visualization similarity judgments to explore significance.
• Innovative Aspects of the Solution:
  • For the first time, domain-agnostic deep learning transfer techniques from computer vision are applied to information visualization.
  • Validate whether deep features can achieve superior similarity judgments across diverse datasets in various visualization tasks.
• Implementation Steps:
  1. Network Architectures and Pre-trained Models: The authors utilized models such as AlexNet, VGG16, ResNet, SqueezeNet, and EfficientNet to extract hierarchical deep features and compute the "perceptual distance" between image pairs.
  2. Experiment Replication: Conducted clustering experiments for scatterplots (based on Pandey et al.'s research) and cognitive experiments on basic visual channels (color, size, shape) by replicating Demiralp et al.'s study.
  3. Evaluation and Comparison: Calculated Spearman rank correlation coefficients in different tasks to evaluate the similarity between deep feature models and human perception results.

Research Findings

Replication of Veras and Collins' Experiment

• Specific Findings:
  • Models such as EfficientNet B0, AlexNet, and SqueezeNet produced clustering labels that outperformed the MS-SSIM method (optimized for specific data), showing high consistency with human scatterplot similarity perception.
  • Deep feature models demonstrated strong performance without requiring any domain-specific training for scatterplots.
• Advantages:
  • Provided a completely domain-agnostic solution relying solely on ImageNet pre-trained weights, reducing the cost and workload of developing customized models.
• Experimental Results:
  • On clustering quality metrics (e.g., AMI, NMI), the average performance of deep feature models was 14.04% higher than MS-SSIM.
• Limitations and Future Directions:
  • The models excel in handling visualizations with complex distributions and spatial structures, such as scatterplots, but their applicability to other data types requires further exploration.

Replication of Demiralp et al.'s Experiment

• Specific Findings:
  • For visual channels related to color, size, and shape, the models performed well in similarity judgments for "size" (close to human judgments) but were weaker in capturing similarity for color and shape.
  • For shape perception, the models lacked the rotational invariance exhibited by humans.
• Advantages:
  • Provided strong evidence that the models perform well in capturing data encoding features, such as spatial differences in object size.
• Experimental Results:
  • In some tasks, model performance approached pixel-space metrics (e.g., MSE), particularly in color channel similarity; the impact of pre-trained weights on low-level visual features was limited.
• Limitations and Future Directions:
  • Deep feature models need further design improvements to address shortcomings in color cultural associations and high-level semantic perception.
  • Extend models to multimodal data visualizations (e.g., scenarios involving text and interactivity); further explore the potential of Stylized ImageNet weights.

Summary and Significance

• Key Findings:
  1. Deep feature-based similarity metrics can effectively work in complex information visualization tasks.
  2. Model performance on basic visual channels is limited by differences between perception and conceptual understanding.
  3. Network architecture has a minor impact on performance, but dataset quality and learning frameworks significantly influence results.
• Future Directions and Applications:
  • Enhance models' rotational invariance and multi-view consistency.
  • Develop unified benchmarks for evaluating similarity models across various visual and non-visual information.
  • Combine deep feature methods to alleviate the workload of human testing in the early and late stages of experimental design (e.g., filtering design options, quantifying design effectiveness).

This research provides valuable insights for the visualization field, demonstrating that deep learning models can further optimize and expand design and analysis tools in information visualization applications.