Learning Custom Experience Ontologies via Embedding-based Feedback Loops

Document Title

Learning Custom Experience Ontologies via Embedding-based Feedback Loops

Document Information

• Subject Area: Human-Computer Interaction, Digital Experience Analysis, Machine Learning for Custom Ontologies
• Keywords: UX Research, Usability Testing, Clickstream Analysis, Sankey Diagram, Sequence Alignment, Embedding-based Feedback Loop, Custom Ontologies, Web Analytics

Research Background and Problem

• What problems or challenges did the authors identify?

  • Existing behavior analysis tools rely heavily on manual labeling and filtering to interpret user interaction data, which is a time-consuming and labor-intensive task.
  • Machine learning methods in existing studies are based on fixed digital experience vocabularies, making them difficult to extend or customize for the needs of different organizations.
  • In large organizations, different teams often use varying terminologies to express similar user intents, leading to challenges in standardizing labels.

• Why is this problem important?

  • Digital experience analysis is a critical component for many enterprises. Automating the labeling process through improved models can significantly reduce time costs and enhance the efficiency and accuracy of data analysis.
  • Standardization and semantic consistency of labels are essential for data reliability, cross-team collaboration, and in-depth analysis.

• Research Motivation and Related Work

  • The authors aim to develop a labeling prediction system that supports the customization of online experience semantics to address the challenges of label standardization and automated customization.
  • Related work includes event tagging tools (e.g., Google Tag Manager) that require manual configuration, as well as techniques for predicting design semantics. While these technologies have seen improvements, they cannot easily extend fixed ontologies to support customization needs.

Solution

• What methods or solutions did the authors propose?

  • The authors proposed a novel interaction model based on embedding-based feedback loops for event label prediction.
  • They utilized general user experience (UX) word embeddings to initialize label predictions and allowed users to correct and customize predictions through interactive visual feedback.

• What is innovative about this solution?

  • The system enhances the vector space through iterative feedback, enabling organizations to define and expand custom semantic vocabularies (e.g., labels), thereby improving the quality of future predictions.
  • It provides the ability to standardize semantics across organizations while supporting teams in developing their own custom ontologies.

• What are the implementation steps and key technologies used?

  • Feedback Loop Components:
    • Frontend: Captures user feedback and displays user navigation paths and intent predictions through interactive Sankey diagrams.
    • Backend: Updates the embedding space based on user feedback, including creating new label embeddings and adjusting vector distances.
  • Embedding Space Initialization:
    • General label embeddings are initialized using language models (e.g., GloVe).
    • The vector space is gradually optimized based on user-defined labels and interaction text.
  • Dynamic Adjustment of Embedding Space:
    • Attraction and repulsion operations are performed to manage semantic distances between vectors.
    • Highly adaptive, capable of handling organization-specific terminology and dynamic language expressions.

Research Outcomes

• What specific results were achieved?

  • The system's deployment in real-world environments demonstrated the efficiency of automated labeling. Users were able to customize labels and use the custom semantics in future tests.
  • The authors showcased the system's generalizability through case studies in two industries, replicating custom labels across multiple rounds of testing on an e-commerce website and a government service website.

• What advantages does it have compared to existing solutions?

  • Unlike models with fixed vocabularies, the embedding-based feedback loop allows for the dynamic expansion of semantic vocabularies.
  • It supports both explicit and implicit user feedback, significantly reducing users' cognitive load.
  • The system can generalize custom label prediction logic across multiple industries and tasks.

• What were the experimental or evaluation results?

  • Experiments showed that over 8 months, the system automatically predicted labels for 15% of click events, with approximately 42% of custom labels successfully appearing in future predictions.
  • Case studies confirmed that a small amount of user-customized feedback could significantly improve the quality of subsequent semantic predictions.

• Limitations and Future Directions

  • Limitations:
    • The current system only predicts intent based on interaction text, without considering visual elements or temporal relationships between screens.
    • It relies on a single language model (GloVe), which may limit the flexibility of semantic predictions.
  • Future Directions:
    • Develop multimodal embedding spaces that incorporate visual elements and temporal context into the prediction framework.
    • Expand the application of language models (e.g., BERT or GPT-3) to enhance semantic representation capabilities.
    • Conduct longitudinal studies to better understand how organizations and teams develop and standardize label ontologies.