DeepSI: Interactive Deep Learning for Semantic Interaction

Document Title

DeepSI: Interactive Deep Learning for Semantic Interaction

Document Information

• Subject Area: Interactive Visual Analytics and Deep Learning
• Keywords: Semantic Interaction, BERT, Visual Analytics, Interactive Deep Learning, Human-Machine Collaboration, Representation Learning, Dimensionality Reduction, Human Perception, Data Projection, Parameter Optimization

Research Background and Problem

• Identified Problem: Current semantic interaction methods rely heavily on the quality of base data representations, and existing deep learning models are only used as feature extractors, failing to adapt to user-specific task requirements, thus limiting their effectiveness.
• Research Importance: Enhancing semantic interaction can more accurately capture analysts' complex cognitive models and reduce the number of user interactions, thereby improving interaction efficiency.
• Motivation and Related Work:
  • Traditional semantic interaction methods (e.g., dimensionality reduction techniques like PCA or t-SNE) perform poorly on nonlinear tasks.
  • Deep learning has demonstrated the ability to automatically extract high-level data representations but is passively used in existing transactional semantic interaction pipelines and is not integrated into the interaction loop.
  • Literature reviews indicate significant potential benefits of integrating semantic interaction with deep learning, but improvements in model design are needed to support user- and task-specific real-time learning.

Solution

• Proposed Method or Solution:

  • A new framework, DeepSI-finetune, is proposed, embedding a BERT-based deep learning model into the semantic interaction process to support dynamic representation learning tailored to users and tasks.
  • A bidirectional interaction loop is designed, where user interactions trigger fine-tuning of the deep learning model, which in turn generates new data features to improve mapping effectiveness.

• Innovations:

  • Achieves real-time fine-tuning of deep learning models, enabling them to adapt interactively to the current analysis task.
  • Optimizes the integration of deep learning and semantic interaction without requiring additional model components, significantly improving interaction efficiency.
  • Utilizes a simple linear dimensionality reduction model based on Weighted Multidimensional Scaling (WMDS), reducing complexity.

• Implementation Steps and Key Techniques:

  • Introduces a bidirectional semantic interaction structure: human interaction (rearranging projection points) and machine learning (updating representations).
  • Employs a distance function-based loss for semantic mapping to quantify the similarity gap between high-dimensional and low-dimensional spaces.
  • Uses a pre-trained BERT model, fine-tuning its parameters through backpropagation.
  • Training efficiency is enhanced by using the Adam optimizer with a low learning rate (e.g., 3e-5).

Research Outcomes

• Specific Results:

  • Proposed the DeepSI-finetune framework, combining BERT and WMDS to support real-time semantic interaction.
  • Demonstrated outstanding performance in experiments through two evaluation methods: a human-centered COVID-19 case study and algorithm-centered quantitative simulations.

• Advantages Compared to Existing Solutions:

  • Compared to the baseline model (DeepSI-vanilla), DeepSI-finetune achieves higher accuracy (task-relevant information capture rate close to 90%), faster response times, and requires fewer user interactions.
  • DeepSI-finetune is capable of handling not only binary classification tasks but also demonstrates significant advantages in three-class and four-class data tasks.

• Experimental or Evaluation Results:

  • Case Study: When analyzing COVID-19-related articles, the projection results accurately clustered four risk factor groups.
  • Simulation Experiments: On three different datasets (SST, Vispubdata, and 20 Newsgroups), DeepSI-finetune's learning curve significantly outperformed DeepSI-vanilla, especially in multi-class tasks.
  • After deeper interaction, DeepSI-finetune was able to adjust representations to better reflect the analyst's target semantics in the projection layout.

• Limitations and Future Directions:

  • The current model has limited interpretability of its internal deep learning states and lacks rich visual feedback to assist users in understanding model behavior.
  • Future work plans:
    1. Develop more interpretable and intuitive visual designs to present model tuning effects.
    2. Explore the use of other loss functions in deep metric learning (e.g., contrastive loss or triplet loss) to replace the current MDS method and further optimize interaction performance.

Summary

By integrating deep learning into the semantic interaction process, DeepSI-finetune significantly improves analysis efficiency and accuracy for user-specific task requirements. Experimental results highlight the potential of interactive deep learning in the field of visual analytics. This work also opens new research directions in interactive deep learning, such as model interpretability and the extension of deep metric learning methods.