Symphony: Composing Interactive Interfaces for Machine Learning

Document Title

Symphony: Composing Interactive Interfaces for Machine Learning

Document Information

• Subject Area: Human-Computer Interaction and Machine Learning Interface Design
• Keywords: Machine learning, AI, visualization, documentation, interactive programming, computational notebooks

Research Background and Problem Statement

• Current machine learning (ML) interfaces, while beneficial for model analysis and data sharing, have low adoption rates, significant usability barriers, and insufficient support for cross-functional team collaboration and reuse. These issues can lead to data errors, model failures, and biases or security problems in ML systems.
• Key issues with existing ML interfaces:
  • Documentation (e.g., Model Cards and Datasheets) often lacks interactive tools and visualization, requiring manual updates.
  • Visualization dashboards are task- and domain-specific, have low reusability, and require users to format data.
  • Interactive programming components (e.g., Streamlit and ipywidgets) are typically limited to specific environments and lack the capability to present complex visualizations.

Importance: The ability to analyze and share ML data and models is crucial for building robust and responsible machine learning systems. Existing tools lack cross-platform and cross-team collaboration efficiency, hindering the establishment of a culture of data insights and sharing.

Research Motivation and Related Work

• Relevant literature includes ML documentation methods (Model Cards and Datasheets), data visualization dashboards (e.g., What-if Tool and ActiVis), and interactive programming environments (e.g., Jupyter Notebook and Streamlit).
• Current research has not achieved cross-platform, modular interactive ML interfaces, nor has it adequately addressed the need for flexibility and sharing in modern team analysis tools.

Solution

• Propose the Symphony framework for composing interactive ML interfaces, supporting task-specific, data-driven components.
• Innovations:
  • Data-driven interactive interfaces that automatically synchronize with ML data and models.
  • Support for task-specific visualizations of complex models and unstructured data (e.g., images, audio, video, sensor data).
  • Modular, reusable, and shareable components with cross-platform support (programming environments and web dashboards).
  • Independent components built using JavaScript, integrating visualization frameworks like Svelte and D3.
• Implementation Steps:
  1. Modular Component Design: Create task-specific interactive visualization components written in Svelte, supporting the generation of analysis tables from raw data and metadata.
  2. Platform Wrappers: Enable seamless transitions between code environments (e.g., Jupyter Notebook) and non-code platforms (e.g., standalone web pages).
  3. Interactive Exploration Tools: Provide user-customizable filtering, grouping, and instance selection tools, with synchronized shared states across components.

Research Outcomes

• Specific Results:
  • Implemented 11 Symphony components, including distribution plots, cross-filters, confusion matrices, projected embedding plots, and bias analysis tools.
  • Deployed in three production ML projects, Symphony helped identify issues such as data duplication, mislabeling, and model blind spots.
  • Enhanced team data insights and cross-functional sharing culture.
• Experiments and Advantages:
  • Comparative experiments demonstrate Symphony's higher efficiency compared to existing tools, with direct usability in core team environments (e.g., notebooks) and rapid generation of shareable documentation.
  • Cross-platform capabilities and task-specific visualizations effectively supported a wider range of unstructured data types.
• Limitations and Future Directions:
  • Limitations:
    • Developing visualization components requires advanced programming skills.
    • Limited to browser memory, making it difficult to handle large-scale datasets.
    • Does not support certain non-data science collaboration platforms.
  • Future Directions:
    • Develop low-barrier tools for component creation.
    • Support more complex interactive guidance, combining exploration with instruction.
    • Extend compatibility to other collaboration platforms, such as instant messaging services or presentation tools.
    • Further research on adaptability to diverse user groups (e.g., small teams or beginners).

Conclusion

The Symphony framework provides powerful tools for ML teams to analyze and share data through interactive, data-driven visualization components with cross-platform support. It fosters collaboration among different roles within organizations. This achievement has the potential to advance ML interfaces and help teams build robust, secure, and responsible AI products.