Erie: A Declarative Grammar for Data Sonification

Title of the Paper

Erie: A Declarative Grammar for Data Sonification

Bibliographic Information

• Subject Area: Data Sonification, Data Visualization Tool Development, Accessible Design
• Keywords: Data Sonification, Declarative Grammar, Accessibility, Assistive Design, Audio Data Charts

Research Background and Problem Statement

• Problems and Challenges:
  • Data sonification maps data variables to auditory variables (e.g., pitch, volume). Despite significant research in this field, existing software support remains limited, making it difficult for researchers to fully explore its potential.
  • Many current tools, such as Sonification Sandbox and other libraries (e.g., R and JavaScript libraries), have limited functionality, often supporting only tightly coupled, specific types of visual charts, which restricts design flexibility.
  • The interfaces of many tools are not programmatic, making them unsuitable for scenarios requiring frequent data updates or user interaction.
• Significance:
  • Data sonification has important applications in accessibility (e.g., assisting visually impaired individuals in understanding data), scientific observation, data art (e.g., exhibitions), and enhancing data storytelling.
  • Providing user-friendly and expressive tools for data sonification design can enable creators to explore diverse designs.
• Research Motivation and Related Work:
  • Existing research in data sonification primarily focuses on the intuitiveness of sound mappings and design methods in accessible visualization.
  • Researchers often rely on manual coding (e.g., Garage Band or other custom solutions), which sets a technical barrier for non-audio professionals.

Proposed Solution

• Method and Framework:
  • The paper proposes a declarative grammar—Erie. This grammar supports abstract expressions of data-to-audio mappings, offering capabilities for rich sonification designs independent of visual forms.
  • Erie allows the definition of extensible pitch designs (e.g., sampling, FM/AM synthesizers) and supports multiple encoding channels, auditory identifiers, and combinations of sonification designs (e.g., sequential playback and layering).
  • A compiler and player for Erie, based on standard Web Audio and Web Speech APIs, are provided.
• Innovations:
  • Independence: Erie is not tied to specific visual chart forms and supports standalone sound chart designs.
  • Expressiveness: It allows the specification of various sound features, channels, and combinations of sonifications (e.g., bass, doubling effects, frequency modulation changes).
  • Data-Driven: The grammar follows a "data-driven" approach, supporting one-click data-to-audio conversion.
  • Extensibility: It supports external audio files, filters, and custom audio node extensions.
  • Compatibility: Compatible with standard audio libraries and browser APIs.
• Implementation Details:
  • Uses a JSON-style syntax to define audio streams, data transformations, pitch, encoding, and stream combinations.
  • Provides playback options (play, pause, seek) and API interfaces for developers to integrate into various audio environments.

Research Findings

• Results and Validation:
  • Erie was used to recreate multiple data sonification prototypes (e.g., Audio Narrative and Chart Reader), demonstrating its ability to support complex sonification designs.
  • The public API and online editor provided by Erie offer developers an accessible platform for tool design.
  • It includes interactive extensions and customization capabilities, showcasing its potential as both a developer tool and a standalone analysis tool.
• Advantages Comparison:
  • Compared to existing tools, Erie offers more audio channels and design combination options (e.g., time, FM/AM modulation, volume, pitch layering).
  • Its declarative grammar supports dynamic audio generation, eliminating the technical barrier of manual coding.
• Experimental or Evaluation Results:
  • Experimental results presented in the paper indicate that designs based on Erie are more reusable, compatible, and flexible to switch. It provides extensible encoder-decoder capabilities and cross-platform API adaptability.
  • Various application scenarios, including real-time density estimation, audio histograms, and model fitting, validate the broad possibilities and technical feasibility offered by Erie.
• Limitations and Future Directions:
  • Limitations: Erie currently targets primarily low-level developer tools, requiring more advanced user experiences (e.g., more support for interactive design and finer-grained playback controls).
  • Future Work:
    • Support for real-time streaming data applications and more dynamic scenarios.
    • Expansion of interaction modes, such as keyboard operations, voice recognition, and haptic feedback for accessibility.
    • Extending Erie to other computational environments, such as R and server-side audio generation.
    • Developing intelligent tools for automated recommendations in data sonification design.

Conclusion

The paper contributes the first declarative grammar for data sonification (Erie) with high extensibility and independence, and validates its potential as a platform for data sonification research and tool development.