Evaluating Singing for Computer Input Using Pitch, Interval, and Melody

Title of the Paper

Evaluating Singing for Computer Input Using Pitch, Interval, and Melody

Paper Information

• Subject Area: Research on human-computer interaction through musical interaction (singing)
• Keywords: musical interaction, pitch, interval, melody, non-verbal input, human-computer interaction, user experience, augmented reality

Research Background and Problem

• Identified Problem: Current voice control systems mostly rely on language commands, which are language-dependent, while non-verbal voice interaction technologies remain underdeveloped.
• Significance: Non-verbal interaction methods can provide language-independent, localized control without requiring complex computational processing, enhancing simplicity and universality in interaction.
• Related Work:
  • Previous studies have explored non-verbal acoustic features such as blowing and vowel sounds as input, but rarely introduced musical concepts like notes and intervals.
  • Research on designing interactions using musical intervals and notes is limited and lacks formal evaluation.

Proposed Solution

• Method: Propose using singing (via pitch, interval, and melody) as a form of input for human-computer interaction.
• Innovations:
  • Introduce musical concepts (seven notes of the major scale) to design interaction methods.
  • Use melody and intervals as the basis for distinguishing commands, enhancing the diversity of input methods.
  • The system allows users to improve singing accuracy by providing reference pitches through background music.
• Implementation Steps:
  1. Propose three basic interaction methods: pitch (single note), interval (two consecutive notes), and melody (sequence of three notes).
  2. Design a visual interface to help users calibrate intervals and melodies.
  3. Provide reference pitch stimuli through background music.
  4. Conduct user experiments with 21 participants to evaluate task efficiency and user experience.
• Technologies Used:
  • Fourier Transform for pitch extraction.
  • Dynamic Time Warping (DTW) algorithm for melody matching.
  • Development of a web-based React experimental application for remote user operation.

Research Outcomes

• Specific Results:
  • Participants completed tasks in an average of 3.5-5 seconds, with post-training error rates ranging from 1.3% to 8.6%.
  • Expert users performed well, but non-singers (novices) had error rates as high as 46%.
  • Background music positively influenced subjective user evaluations but had limited impact on overall task time and error rates.
• Advantages:
  • Compared to traditional voice-based interaction, musical interaction methods are language-independent.
  • Opens a new research direction in non-verbal input methods.
  • The introduction of musical intervals and melodies enriches the types of system interaction commands.
• Experimental Evaluation Results:
  • User feedback indicated that the interaction design was mostly enjoyable, especially for users with a musical background who adapted quickly.
  • Error rates were highly correlated with user background (musical training, daily use of technology).
• Limitations and Future Directions:
  • Steep learning curve for beginners, with poor experience for non-singers.
  • Although background music received positive subjective feedback, it had limited impact on task performance; future research could explore alternative background music designs.
  • Advanced interaction tasks (e.g., melody) caused user fatigue, requiring further optimization of task complexity.

Application Scenarios

• Supplementing Traditional Input Methods:
  • Use musical input (e.g., pitch, melody) as shortcuts for applications.
  • Quickly switch tools or adjust parameters in drawing applications.
• Independent Input Methods:
  • In augmented reality (AR) devices, use singing to select colors or create 3D blocks.
  • Provide effective support for scenarios without keyboards or mice.
• Scenarios Without Visual Feedback:
  • Applicable in driving scenarios (e.g., adjusting in-car environment parameters) or controlling music and volume via headphones.
• Supporting Accessibility Design:
  • Design singing interaction devices for users with motor or speech impairments.

Through this research, the authors provide new possibilities for human-computer interaction technologies, demonstrating the potential of singing-based interaction and how it can support diverse usage scenarios. The study also highlights directions for optimizing user groups and interaction design in future research.