PSST: Enabling Blind or Visually Impaired Developers to Author Sonifications of Streaming Sensor Data

Title of the Paper

PSST: Enabling Blind or Visually Impaired Developers to Author Sonifications of Streaming Sensor Data

Paper Information

• Domain: Accessibility technology and physical computing, supporting blind or visually impaired developers in creating tools for sonifying real-time data.
• Keywords: Accessibility technology, physical computing, sound technology, data visualization, blind developers, data streams, programming tools, dynamic data, sonification tools, data sensors
• Conference: The 35th Annual ACM Symposium on User Interface Software and Technology (UIST ’22)

Research Background and Problem Statement

• Identified Issues or Challenges:
  • Current physical computing tools and programming environments predominantly assume users have visual capabilities, lacking support for blind and visually impaired users in accessing real-time data.
  • Sensor data states are typically presented visually, making it difficult for blind developers to debug and understand sensor-generated data using existing tools.
  • Existing sonification tools primarily focus on static data and do not support sonification or interaction with streaming data.
• Significance:
  • Enhancing the interaction capabilities of blind developers with physical computing technologies can promote diversity in STEM (Science, Technology, Engineering, and Mathematics) education and outreach.
  • Providing real-time data comprehension tools for users who rely entirely on auditory feedback can assist in testing and debugging systems and their sensors.
• Research Motivation and Related Work:
  • Although prior research has attempted to optimize accessibility technology through participatory design, little attention has been given to helping blind users dynamically understand streaming data.
  • Needs analysis indicates that blind developers urgently require a tool for flexible development and dynamic debugging of sonified real-time data.

Proposed Solution

• Method or Solution:
  • A tool named PSST (Physical computing Streaming Sensor data Toolkit) is proposed to support blind or visually impaired developers in creating sonifications of streaming data.
  • PSST offers customizable data transformation, filtering, and multimodal output functionalities, enabling users to design dynamic data sonifications.
  • A dedicated graphical user interface (PSST Dashboard) is designed to facilitate use by blind users without programming backgrounds.
• Innovative Aspects of the Solution:
  • Through accessible sonification and screen reader-friendly settings, BVI (Blind/Visually Impaired) developers can independently create personalized data presentations.
  • Supports real-time sonification of streaming data and accommodates various types of input, output, and data processing.
  • Provides multiple interaction modes for sensor data, such as melodic pitch representation, alert sounds, and range-based filtering.
• Implementation Steps and Key Technologies:
  • Core Architecture: Includes modules for data input (DataSink), data processing (DataHandler), and data output (DataOutput).
  • Technical Implementation: Built using TypeScript, leveraging the Web Audio API and Web Speech API for sound output, and RxJS for processing data streams.
  • Visualization Interface: Offers a graphical user interface accessible via screen readers.

Research Outcomes

• Specific Results:
  • Designed and implemented the PSST toolkit and PSST Dashboard, supporting BVI users in achieving multimodal data presentation.
  • With PSST, users can perform real-time sonification of streaming data, parallel processing of multiple sensors, and dynamic debugging.
  • Validated through a series of application scenarios, such as simulating mouse position, high-pitch control for keyboard interaction, and generating data logs from physical sensors.
• Advantages:
  • Compared to existing sonification tools, PSST emphasizes the customization needs of blind users, such as dynamically configuring different pitches, voices, and sound types.
  • Demonstrated the practicality of PSST in real-time monitoring of sensor responses, analyzing trends, and handling dynamic interactive data.
• Experimental and Evaluation Results:
  • User studies involving two blind developer participants validated the toolkit’s usability and accessibility, with users expressing satisfaction with its customization and real-time feedback capabilities.
  • Users learned sensor response patterns, identified anomalous sensor values, and explored interactions with multi-sensor data streams.
• Limitations and Future Directions:
  • The current PSST database functionality is limited, and supported audio output types are restricted (e.g., no support for dynamic panning or stereo output).
  • The user experience of BVI developers using the toolkit’s programming interface has not been thoroughly explored.
  • Future work may expand the toolkit to other data-intensive domains (e.g., machine learning and data science) and introduce timeline playback functionality.