Starrypia: An AR Gamified Music Adjuvant Treatment Application for Children with Autism Based on Combined Therapy

Title of the Paper

Starrypia: An AR Gamified Music Adjuvant Treatment Application for Children with Autism Based on Combined Therapy

Paper Information

• Subject Area: Adjuvant therapy for children with autism, integrating Augmented Reality (AR), music therapy, and deep learning technologies
• Keywords: Children with autism, augmented reality, serious games, deep learning, behavioral intervention, music therapy

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Autism is characterized by limited verbal communication, abnormal social interactions, and rigid behavioral patterns. Existing treatment methods (e.g., Applied Behavior Analysis (ABA), music therapy, and sensory integration training) are often singular in approach, lack appeal, and have limited efficacy.
  • High treatment costs and uneven distribution of medical resources (e.g., in China, there are 3-5 million children with autism but only about 2,000 rehabilitation institutions) restrict accessibility to therapy for children with autism.
  • Music therapy lacks a targeted music database and innovative multimodal interaction scenario designs, particularly those centered on children's interests and abilities.

• Significance:

  • Early diagnosis and intervention during childhood are crucial for improving the quality of life for children with autism.
  • There is a need for a low-cost, efficient, and engaging digital therapeutic approach that is widely accessible and minimally constrained.

• Research Motivation:

  • To combine the advantages of AR technology, multimodal interactive games, and music therapy to develop an effective intervention tool for autism that alleviates symptoms while catering to children's interests and engagement.

Solution

• Proposed Method or Solution:

  • Developed an AR gamified music adjuvant treatment application, "Starrypia," based on the ABA behavioral theory framework. This application innovatively integrates ABA, music therapy, and sensory integration training to provide a lightweight, engaging, and effective digital therapeutic tool.
  • Deep learning (BiLSTM model) is employed to generate music segments tailored to autism intervention needs, addressing the scarcity of therapeutic resources.

• Innovations:

  • First integration of ABA behavioral theory, music therapy, and sensory integration training into a single application, embedding behavioral interventions into the game design process.
  • First application of deep learning to generate customized music for autism music therapy.
  • Realized a multimodal interactive experience, including 3D visual scenes, touch interaction, and music generation, to enhance children's engagement and therapeutic outcomes.

• Implementation Steps and Technical Details:

  1. UI Design: Tailored to the psychological characteristics of children with autism, using blue as the primary color (most favored by children), simplifying the interface, and incorporating intuitive cartoon characters.
  2. Operational Flow and Level Design:
     • Designed game levels based on the core principles of ABA: stimulus, response, reinforcement, and pause.
     • Divided music therapy into reception, improvisation, and recreation, conducting interventions through multisensory stimulation (visual, auditory, tactile).
     • Introduced specific scoring mechanisms (e.g., performance scores, usage time) to evaluate operational effectiveness.
  3. Music Generation:
     • Generated harmonious music based on MIDI datasets (including the Nottingham database and 100 children's songs) and the BiLSTM model.
     • Arranged main melodies, chords, and embellishments, outputting playable music files.
  4. Evaluation Module:
     • Recorded children's performance and provided feedback using a "Music Diary" and "Evaluation Module."
  5. Experimental Validation:
     • Conducted a 4-week controlled experiment involving 20 children with autism.

Research Outcomes

• Specific Results:

  • "Starrypia" effectively improved children's sensory abilities, verbal communication, and self-help skills, with the most significant improvement observed in sensory abilities.
  • Music generated by the BiLSTM model demonstrated high consistency in style and fluency with original music, showing substantial therapeutic potential.
  • The design of the scenes and interface appeal was validated, with most children showing noticeable interest and high engagement levels.

• Advantages Over Existing Solutions:

  • By innovatively integrating traditional intervention methods, the solution not only enhanced therapeutic outcomes but also significantly improved children's user experience.
  • Leveraged AR technology and deep learning to overcome the spatial and temporal limitations of traditional therapies.
  • Lightweight treatment costs made the solution accessible to more families.

• Experimental or Evaluation Results:

  • The average ABC score decreased by 3 points (treatment group), outperforming the control group that did not receive Starrypia intervention (decrease of 1.3 points).
  • The average CARS score decreased by 1.5 points (treatment group), compared to a decrease of only 0.6 points in the control group.
  • Behavioral observations indicated that children were more focused and proactive, with some even exhibiting social and sharing tendencies.

• Limitations and Future Directions:

  1. Limitations:
     • Small sample size of only 20 children, with individual differences and a short duration of 4 weeks, resulting in insufficiently significant statistical outcomes.
     • The therapy may not be suitable for children with high sensory sensitivity, and the diversity of characters and levels needs improvement.
     • Long-term user retention has not been verified, and prolonged exposure to electronic devices may pose health risks.
  2. Future Directions:
     • Expand the sample size and experiment duration to obtain more stable and significant statistical results.
     • Increase the variety of game characters, levels, and personalized settings (e.g., customization of colors and background music).
     • Introduce mechanisms to control game duration to reduce the risk of overuse.
     • Explore real-time quantification methods for therapeutic effects, such as incorporating attention tests for children.