Structure-aware Visualization Retrieval

Document Title

Structure-aware Visualization Retrieval

Document Information

• Subject Area: Data Visualization Retrieval and Representation Learning
• Keywords: Data Visualization, Visualization Retrieval, Visualization Similarity, Representation Learning, Visualization Embedding

Research Background and Problem

• With the widespread use of data visualization, a large number of visualizations in scalable vector graphics (SVG) format are being created and shared online. However, retrieving perceptually similar graphics from large-scale visualization libraries has become a prominent research challenge.
• Existing methods primarily treat visualizations as bitmap images, focusing only on visual appearance while neglecting the inherent structural information of the SVG format, including spatial and hierarchical relationships of visual elements. This structural information can provide a comprehensive representation of visualizations from a new perspective.
• Challenge: How to effectively process and integrate visual and structural information to improve visualization retrieval performance.

Solution

• Method: A perceptually structure-sensitive visualization retrieval method is proposed, combining visual and structural information to evaluate the perceptual similarity of visualizations.
  • Graph representation learning techniques are employed to convert SVGs into graph models reflecting structural information, while visual information is transformed into bitmaps.
  • Contrastive representation learning techniques are applied to generate low-dimensional embedding vectors through self-supervised learning, representing both visual and structural information.
  • Convolutional Neural Networks (CNNs) are used to extract visual information, while Graph Neural Networks (GNNs) extract structural information. The two types of information are fused through vector concatenation.
• Innovations:
  • For the first time, the joint consideration of structural and visual information is explicitly proposed to quantify the perceptual similarity of visualizations.
  • An efficient self-supervised contrastive learning method is adopted, reducing the burden of data annotation.
  • The traditional retrieval methods based on visual features are extended to handle structural data in SVG format.
• Implementation Steps:
  1. Structural Information Extraction: Extract element features (e.g., position, type, style) from SVGs and construct graph representations of visual elements.
  2. Visual Information Extraction: Use a contrastive learning model (SimSiam) to generate visual representations.
  3. Information Fusion: Normalize and concatenate the embedding vectors of visual and structural information, and perform retrieval based on cosine distance.

Research Outcomes

• Specific Outcomes:
  • Proposed embedding vectors to represent the perceptual similarity of visualizations, successfully enabling similarity-based visualization retrieval.
  • Designed and implemented a structure-sensitive retrieval process, significantly improving retrieval performance.
• Advantages and Comparisons:
  • Compared to traditional methods that only consider visual information (e.g., CNN or HOG features), the proposed method demonstrates superior performance in terms of type consistency, element count consistency, and trend consistency.
  • Experiments and user studies validate the method's advantages in retrieval accuracy and subjective user evaluations.
• Experimental Results:
  • Quantitative analysis shows that retrieval methods incorporating structural information outperform baseline models across various performance metrics.
  • User studies confirm a significant improvement in user satisfaction with the retrieval results (average score of 2.86, higher than other methods).
  • Case studies further validate the method's significant advantages in distinguishing chart types and ensuring trend consistency.
• Limitations and Future Directions:
  • The current retrieval method does not incorporate quantitative metrics for trends; future work should explore unified quantification schemes for visualization trends.
  • The method currently offers limited support for other formats (e.g., non-SVG graphics); future work could extend it to bitmap graphics and multi-view visualization scenarios.
  • The embedding vector fusion strategy may involve redundant information and storage efficiency issues, necessitating the development of more efficient feature fusion methods.