Generative Role-Play Communication Training in Virtual Reality for Autistic Individuals: A Study on Job Coach Experiences in Vocational Training Programs

Research Background and Issues

• What problems or challenges did the authors identify?
  The authors pointed out that traditional soft skills training for individuals with autism involves significant demands, requiring job coaches to spend extensive time creating personalized content and repeatedly conducting one-on-one role-playing sessions. At the same time, existing methods face challenges in adopting advanced technologies, such as virtual reality (VR) and large language model (LLM)-based tools, due to factors like insufficient technical proficiency among coaches and user-unfriendly technological environments.

• Why is this issue important?
  Improving soft skills for individuals with autism can significantly enhance their employability. Enhancing the usability of technological tools not only alleviates the burden on job coaches but also provides effective and efficient training support to a broader population of individuals with autism.

• Research Motivation and Related Work
  Building on prior research, the authors explored the potential advantages of VR and AI-driven solutions, such as low cost, high adaptability, and safe training environments. However, most existing studies focus on the trainees' experience while overlooking the challenges faced by job coaches. This study aims to evaluate the feasibility and practical challenges of integrating an LLM-driven VR platform from the perspective of job coaches.

Solution

• What methods or solutions did the authors propose?
  The authors developed a VR and LLM-based soft skills training platform designed to support job coaches in creating and conducting soft skills training for individuals with autism. The platform includes a VR application equipped with LLM-driven virtual characters and a web interface that allows coaches to generate role-playing scenarios via text input.

• What are the innovative aspects of the solution?
  The platform integrates LLM and VR technologies, enabling the generation of diverse and highly adaptive training scenarios while providing an easy-to-use interface for job coaches to design role-playing content. This design lowers the technical barriers and introduces the concept of generative technologies into autism training applications.

• What are the implementation steps and key technologies used?

  1. Platform Content Framework: The web interface supports scenario creation (inputting titles, role labels, and scenario instructions) and dialogue review.
  2. Core Technologies: The platform utilizes OpenAI's GPT-3.5 model and VR environments to generate role-playing dialogues adapted to real-world scenarios.
  3. System Integration: A server connects the web interface and VR application, ensuring synchronization of scenario information and training session data.
  4. Training Process: Coaches create scenarios, trainees engage in role-playing through VR, and all interactions are recorded on the platform for subsequent evaluation.

Research Outcomes

• What specific results were achieved?
  The authors conducted a three-month longitudinal study in collaboration with 13 job coaches, validating the platform's feasibility through guided training sessions (Phase 1) and feedback interviews based on long-term use (Phase 2). Over the three months, a total of 12 scenarios were generated, and 151 scenario sessions were conducted.

• What advantages does it have compared to existing solutions?

  • Customized Scenarios: Coaches can generate scenario content tailored to trainees' needs, enhancing the personalization of training.
  • Immersive Experience: The platform provides an immersive training environment through VR, effectively improving trainee engagement and confidence.
  • Ease of Use: The optimized design enables job coaches to easily master LLM prompt-writing techniques.

• What were the experimental or evaluation results?

  • Job coaches generally found the platform user-friendly and powerful but mentioned some technical and usability challenges (e.g., setup time, screen projection complexity).
  • Trainees with autism demonstrated high engagement during use, though certain challenges (e.g., speech recognition issues, comfort for glasses wearers) require further attention.
  • Coaches observed that trainees gradually gained confidence and were able to interact independently with virtual characters.

• Limitations and Future Directions

  • Limitations: The study primarily focused on the perspective of job coaches without delving deeply into the long-term transfer of skills by trainees in real-world environments. Additionally, equipment and scenario limitations may affect training quality.
  • Future Directions:
    1. Explore optimized use of different VR devices.
    2. Enhance training scenarios and virtual characters with higher realism.
    3. Consider hybrid applications (e.g., desktop versions) to address the personalized needs of trainees with autism.
    4. Expand to hard skills training modules to further enhance the functionality of VR tools in vocational training.

In summary, this study highlights the unique value of an LLM-driven VR platform in the field of vocational training for individuals with autism while revealing some bottlenecks in its sustainable integration. The research provides valuable design guidelines and improvement suggestions for future technological development in this domain, contributing significantly to advancing technology integration in the field.