PTeacher: a Computer-Aided Personalized Pronunciation Training System with Exaggerated Audio-Visual Corrective Feedback

Document Title

PTeacher: a Computer-Aided Personalized Pronunciation Training System with Exaggerated Audio-Visual Corrective Feedback

Document Information

• Subject Area: Human-Computer Interaction, Language Learning, Computer-Aided Pronunciation Training
• Keywords: Computer-Aided Pronunciation Training System, Audio-Visual Corrective Feedback, Language Learning, Exaggerated Feedback, User Study

Research Background and Problem

• Problem/Challenge: Second language (L2) English learners face difficulties in improving pronunciation, especially in the absence of expressive and personalized corrective feedback. Existing Computer-Aided Pronunciation Training (CAPT) systems primarily focus on natural speech synthesis and comparing correct and incorrect pronunciations but lack feedback strategies that are easily perceivable and distinguishable.
• Significance: Pronunciation ability is a critical factor in language learning, but current resource limitations and the increasing age of learners reduce learning efficiency, particularly in underdeveloped regions where English teaching resources are scarce.
• Research Motivation and Related Work: Exaggerated feedback has been proven effective in traditional teaching, but existing CAPT systems lack systematic research on the "appropriate degree of exaggeration." Moreover, how to tailor feedback based on learners' English proficiency remains underexplored.

Solution

• Method or Solution:
  • Propose a CAPT system named "PTeacher" that corrects learners' pronunciation through exaggerated audio and visual feedback while designing a personalized dynamic feedback mechanism.
  • Integrate real-time Mispronunciation Detection and Diagnosis (MDD) algorithms to provide exaggerated feedback tailored to learners' proficiency levels.
  • Develop interactive pronunciation training courses to enhance user engagement.
• Innovations:
  • Define precise exaggeration parameters for audio and visual feedback, ensuring feedback is identifiable, comprehensible, and perceptually effective through user-participation studies.
  • Introduce a personalized dynamic feedback mechanism that adjusts the degree of exaggeration based on learners' English proficiency.
  • Provide lifecycle pronunciation ability assessments and comprehensive learning improvement reports.
• Implementation Steps and Key Technologies:
  1. Use the MDD algorithm to perform real-time detection and diagnosis of pronunciation at the syllable, word, and sentence levels.
  2. Employ Text-To-Speech (TTS) technology to generate neutral speech and adjust the pitch, duration, and energy of selected phonemes using an exaggeration generator.
  3. Simulate the movements of speech organs visually and enhance auxiliary diagrams with artistic design, such as increased color saturation and directional cues.
  4. Offer Interactive Participatory Drama and Custom Courses to stimulate greater learner interaction.

Research Outcomes

• Specific Outcomes:
  • Determine the optimal exaggeration parameters for the audio exaggeration generator and learners' preferences for suitable visual exaggeration levels.
  • Develop pronunciation training courses capable of dynamically adjusting feedback, improving learner engagement and learning experience.
  • Validate the system's significant improvement in learners' learning efficiency through comparative experiments.
• Advantages Over Existing Solutions:
  • The PTeacher system improved learners' pronunciation accuracy by an average of 14.19% (advanced learners) and 27.55% (beginner learners), significantly outperforming systems without exaggerated feedback or non-personalized feedback.
  • Compared to systems with expert guidance, the PTeacher system achieves comparable learning outcomes while offering broader applicability and convenience.
• Experimental or Evaluation Results:
  • In user participation experiments, users demonstrated significant improvements in perceiving and understanding audio and visual feedback.
  • The system not only enhanced users' ability to learn English but also increased their interest in learning through interactive courses.
  • The system showed potential in addressing educational inequities in low-resource environments by supplementing the lack of foreign teaching resources.
• Limitations and Future Directions:
  • Current MDD algorithm misjudgments (false acceptance or rejection) may affect feedback accuracy, though the impact is limited.
  • The design of exaggerated feedback still relies on manual parameter definition; future work could leverage deep learning techniques to automate parameter design.
  • The system currently supports only two proficiency levels; future iterations could introduce more granular feedback levels.

Summary and Discussion

• The PTeacher system presents an effective new approach in the field of computer-aided language learning, particularly in pronunciation training, by providing users with personalized exaggerated feedback.
• The system's design principles and technologies could be extended to other imitation learning scenarios, such as dance or piano instruction, though they may not be applicable to logical deduction tasks like mathematics teaching.
• The system emphasizes the importance of enhanced user perception and personalized feedback, with extensive user experiments validating its advantages. Future work could focus on further improving system intelligence and expanding its functional coverage.