"Hey Dashboard!": Supporting Voice, Text, and Pointing Modalities in Dashboard Onboarding using Large Language Models

Paper Title

"Hey Dashboard!": Supporting Voice, Text, and Pointing Modalities in Dashboard Onboarding using Large Language Models

Publication Info

• Topic area: Multimodal onboarding systems for visualization dashboards using large language models.
• Keywords: Dashboard onboarding, multimodal interaction, large language models, voice-based interaction, visual highlights, contextual help, Microsoft Power BI, user study, human-like assistance, exploratory tasks.

Background and Problem

• Problem / challenge: Existing dashboard onboarding methods, such as static guides and tutorials, are often outdated, non-interactive, and fail to adapt to individual user needs or preferences. Automated solutions lack the contextual and multimodal capabilities of human experts.
• Significance: Effective onboarding is critical for enabling users to navigate and analyze complex dashboards, reducing barriers to adoption and improving usability in organizational decision-making.
• Motivation and related work: Prior work has explored static onboarding methods (e.g., guided tours, annotations) and multimodal interaction for data visualization, but these approaches remain limited in interactivity and adaptability. This paper builds on Dhanoa et al.'s process model for dashboard onboarding, aiming to bridge the gap between static solutions and dynamic, human-like assistance.

Solution

• Proposed approach: Diana (Dashboard Interactive Assistant for Navigation and Analysis), a multimodal onboarding assistant powered by large language models, integrates voice, text, mouse-based interactions, and visual highlights to provide dynamic, in-situ guidance.
• Novelty:
  1. Introduction of a multimodal onboarding assistant combining voice, text, and mouse-based interactions.
  2. Implementation of contextual help menus and visual highlights for user orientation within dashboards.
  3. Integration of large language models to provide human-like onboarding support tailored to specific dashboard contexts.
• Procedure and key techniques:
  • Diana supports three interaction modalities: voice (push-to-talk), keyboard (chat), and mouse (lasso selection).
  • Feedback is provided through text, audio, contextual menus, and visual highlights.
  • A radial menu organizes onboarding information hierarchically into categories like "Read," "Data," "Interact," and "Insight."
  • The system uses metadata from Microsoft Power BI dashboards and OpenAI APIs for natural language processing and speech synthesis.

Results

• Concrete findings: Participants using Diana achieved higher task accuracy and completeness compared to those relying on static dashboard guides. Voice-based interaction and visual highlights were particularly effective, with participants expressing trust and satisfaction in the system.
• Advantage over baselines: Diana users solved more tasks correctly and expressed less uncertainty, while baseline participants often relied on manual calculations and struggled with dashboard interactivity.
• Experiments / evaluation:
  • Mixed-methods user study with 12 participants (6 using Diana, 6 using static guides).
  • Tasks varied by type (lookup, exploratory, interpretive), cardinality (single/multiple views), and difficulty (easy, medium, hard).
  • Data collected through pre-questionnaires, think-aloud sessions, and post-task interviews.
• Limitations and future work:
  • Limited adoption of the radial menu due to perceived complexity and unfamiliarity.
  • Study conducted on a single dashboard, limiting generalizability.
  • Future work includes redesigning the radial menu, extending Diana to other dashboard platforms, and conducting longitudinal studies to measure learning improvement.

Summary

Diana is a multimodal dashboard onboarding assistant that combines voice, text, mouse-based interactions, and visual highlights to provide dynamic, human-like guidance. In a user study, Diana outperformed static guides in task accuracy and user satisfaction, with voice-based interaction emerging as a preferred modality. The system leverages large language models to adapt to user queries and dashboard contexts, fostering autonomy while maintaining usability. Future work will focus on improving the radial menu design, expanding platform compatibility, and conducting extended evaluations to validate its effectiveness across diverse dashboards.