"As an Autistic Person Myself:" The Bias Paradox Around Autism in LLMs

Research Background and Issues

• What problems or challenges did the authors identify?
  The authors pointed out that although large language models (LLMs) like ChatGPT are widely applied in disability research, particularly in autism-related fields, the potential biases regarding autism within these models have not been thoroughly investigated. Rapidly disseminated models may influence public perceptions of marginalized groups, such as autistic individuals, on both implicit and explicit levels.

• Why is this issue important?
  Autistic individuals represent a systematically overlooked group, with a significant global prevalence, yet societal understanding and acceptance remain limited. As LLMs are increasingly utilized in society, these models may reinforce existing biases or provide inaccurate perceptions, negatively impacting social and technological development.

• Research Motivation and Related Work
  This study aims to explore implicit and explicit biases related to autism within ChatGPT. It integrates preferences for identity-first language (emphasizing autism as part of individual identity) and existing research on autism biases, advancing the understanding of marginalized groups within models and exploring ethical considerations in technology.

Solutions

• What methods or solutions did the authors propose?
  The authors employed an experimental method based on "persona prompting," selecting a persona representing an autistic individual generated by ChatGPT and analyzing the model's behavior and logic behind this selection.

• What is innovative about this solution?
  The authors applied the philosophical concept of the "bias paradox" to the LLM domain, demonstrating the inherent tension within these models as they attempt to simultaneously express mainstream perspectives (e.g., viewing autism through a deficit lens) and marginalized viewpoints (e.g., considering autism as part of diversity). This approach reveals potential mechanisms underlying fundamental challenges in model development.

• What are the implementation steps and key technologies used?

  1. Write Python scripts to call GPT-3.5 via API, generating three virtual agent personas with assigned attributes (age, occupation, personality, etc.).
  2. Instruct GPT to select one persona as autistic, provide reasoning, and revise the attribute descriptions.
  3. Analyze data from a total of 800 experiments, conducting quantitative analysis (e.g., the impact of gender, age, occupation on selection) and qualitative analysis (extracting stereotypes or implicit views about autism).
  4. Compare and interpret the internal contradictions and biases in the model's descriptions of autistic personas.

Research Outcomes

• What specific outcomes were achieved?

  • Quantitative Analysis: The model tends to select young males as representatives of autism, with occupation significantly influencing the choice—technical or analytical roles such as software engineers are more frequently associated with autism. Interestingly, older personas were also occasionally identified as autistic, showing no significant correlation with age.
  • Qualitative Analysis: Four notable bias patterns regarding autism in ChatGPT were revealed:
    1. Difficulties in social interaction (perceived as "social awkwardness").
    2. Heightened sensory sensitivity requiring specific management.
    3. Autism endowing individuals with "unique" talents primarily serving others.
    4. Autistic individuals needing additional support and community assistance to succeed in life.

• What advantages does this solution have compared to existing ones?
  This study not only uncovers explicit biases but also delves into implicit biases, proposing ways to interpret the tension within LLMs regarding the same topic (e.g., perspectives on autism). This approach provides a novel lens for bias analysis and expands research methodologies in the HCI field.

• What are the experimental or evaluation results?
  The study found that ChatGPT attempts to promote inclusivity (e.g., emphasizing the importance of diversity) while simultaneously reflecting typical stereotypes about autism, such as social difficulties and occupational limitations. This highlights the foundational biases in model training data and the developers' conflicting adjustments for inclusivity.

• Limitations and Future Directions

  • Limitations:
    1. The study only used GPT-3.5, which may not apply to other LLM versions.
    2. Non-binary gender personas were excluded, and the impact of intersectional identities (e.g., race or sexual orientation) on biases was not considered.
    3. The experimental environment was simplified and did not include data or feedback from real autistic communities.
  • Future Directions:
    1. Test differences in autism biases across various LLMs.
    2. Include real autistic users (with diverse gender and cultural backgrounds) to provide more nuanced and effective analysis.
    3. Explore how autism biases influence other domains (e.g., algorithmic discrimination in recruitment processes).
    4. Enhance LLMs' ability to express self-stance and uncertainty in generated outputs.