Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction

Title of the Paper

Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction

Paper Information

• Subject Area: Visualization, Multimodal Interaction, Personal Data
• Keywords: Personal Informatics, Data Visualization, Multimodal Interaction, Speech, Smartphones

Research Background and Problem

• Problem:

  • Most existing mobile health applications provide limited support for navigating and exploring personal health data, particularly in handling temporal dimensions.
  • Research on data visualization on mobile devices (especially smartphones) is scarce, facing challenges such as limited screen space and low input precision.
  • Interaction methods supported by traditional mouse devices (e.g., hover, detailed information display) are difficult to implement on mobile devices.

• Significance:

  • As smartphones become the primary devices for daily information access, there is an increasing need for efficient and convenient exploration of personal health data.
  • Flexible navigation of temporal data (e.g., specific dates or time ranges) can help users gain deeper insights and self-reflection on their health data.

• Research Motivation and Related Work:

  • Previous studies have shown that multimodal interaction (e.g., combining speech and touch) can enhance user experience.
  • To address the limited temporal operation support in most current mobile health applications, introducing speech interaction can expand flexibility and expressive freedom.

Solution

• Method or Solution:

  • Proposed and developed Data@Hand, a smartphone application supporting speech and touch interaction for exploring personal health data.
  • The application integrates a speech framework (Apple or Microsoft Speech API) and Fitbit REST API to retrieve users' health data, enabling navigation, temporal comparisons, and data queries through multimodal interaction combining speech and touch.

• Innovations:

  • For the first time, combines speech and touch interaction on mobile devices (particularly smartphones) to enable complex data exploration functions.
  • Provides flexible temporal operations, such as specifying time ranges using natural language (e.g., "last Thanksgiving").
  • Maintains simple and intuitive graphical visualization forms within limited screen space while supporting detailed temporal comparisons.

• Implementation Steps and Technology:

  • User interface design includes navigation pages, detailed data source pages, and two types of temporal comparison pages (dual-range comparison and periodic comparison).
  • Offers three interaction modes: touch-only, speech-only, and a combination of touch and speech.
  • Data queries are implemented through natural language commands (e.g., "days with more than 10,000 steps"), with results highlighted.
  • The application is implemented using TypeScript and React Native, compatible with iOS and Android platforms.

Research Outcomes

• Specific Outcomes:

  • Conducted exploratory experiments to investigate the advantages of combining speech and touch interaction in information visualization.
  • Participants were able to flexibly and smoothly explore personal health data, navigate, compare, and query through multimodal interaction.
  • Designed aggregation plots tailored for small-screen devices, efficiently presenting data with averages and ranges during temporal comparisons.

• Advantages over Existing Solutions:

  • Compared to mobile applications relying solely on touch, Data@Hand offers more flexible temporal navigation operations.
  • The side-by-side comparison of data from different time periods enhances perception of trends and differences without relying on memory to switch between pages.
  • The combination of speech and touch simplifies operational steps and improves precision.

• Experimental or Evaluation Results:

  • A total of 13 long-term Fitbit users participated in the experiment, reporting positive experiences, with most expressing a desire to continue using the application.
  • Quantitative results showed that users successfully completed various data operations (temporal navigation, comparison, query) using multimodal interaction on average.
  • Qualitative analysis revealed that users gained rich personal insights through Data@Hand, such as behavioral comparison trends, data extremes, and temporal patterns.

• Limitations and Future Directions:

  • Limitations:
    • Current speech recognition technology has limited capabilities in handling personal events (e.g., "graduation ceremony" or "summer vacation").
    • Highlighting query results is less effective in long-term data aggregation views.
    • Usage in public spaces may be constrained by privacy concerns and environmental interference.
  • Future Directions:
    • Enhance support for speech commands by adding user-defined semantic tags (e.g., personal significant events).
    • Explore more efficient visualization methods, particularly for representing long-term data.
    • Extend to other domains, such as productivity, financial history, and other personal data exploration scenarios.

Conclusion

Data@Hand integrates complementary speech and touch interactions to optimize personal data exploration in smartphone environments, providing a novel and practical solution for the field of mobile visualization.