AutoML in The Wild: Obstacles, Workarounds, and Expectations

Title of the Paper

AutoML in The Wild: Obstacles, Workarounds, and Expectations

Paper Information

• Research Domain: Automated Machine Learning (AutoML), Human-Computer Interaction (HCI), Privacy Protection and Customization Technologies
• Keywords: AutoML, Privacy, Transparency, Customization, User Autonomy

Research Background and Issues

• Identified Problems or Challenges:

  1. AutoML faces issues such as lack of transparency, insufficient customization, and privacy risks.
  2. Current research primarily focuses on technical aspects, lacking exploration of user dynamics and coping strategies in real-world applications.

• Significance: AutoML aims to lower the barrier to using machine learning technologies, making data science and decision-making more accessible. However, design shortcomings in complex real-world scenarios may hinder its adoption and user experience.

• Research Motivation and Related Work:

  1. Existing studies on AutoML mainly concentrate on enhancing human trust, improving tool transparency, and refining process designs.
  2. This paper seeks to delve into user perspectives, exploring needs and practices in real-world applications and providing solutions to overcome imperfections in the technology.

Proposed Solutions

• Methods or Solutions Proposed: Through semi-structured interviews, the authors analyzed user strategies for addressing AutoML's deficiencies (lack of transparency, customization, and privacy concerns) and summarized selective and specific usage methods employed by users.

• Innovations:

  1. Introduced the concept of "user autonomous work," emphasizing how users actively create coping strategies to address technological shortcomings.
  2. Highlighted privacy issues, which are rarely discussed in the AutoML domain.
  3. Conducted a socio-technical perspective study, providing an in-depth analysis of users' actual work contexts, needs, and behavioral patterns.

• Implementation Steps and Key Techniques:

  1. Conduct semi-structured interviews to collect users' experiences and reasons for strategic use of AutoML.
  2. Analyze data using thematic analysis to identify major user challenges and coping strategies.
  3. Extract design insights to guide future optimization of AutoML platforms.

Research Outcomes

• Specific Findings:

  1. Identified three major limitations of AutoML: lack of transparency, insufficient customization, and privacy risks.
  2. Users employed various strategies, including customizing input data, developing internal AutoML tools, manually inspecting model results, and choosing reputable platforms.
  3. Users selectively utilized AutoML based on specific contexts to meet performance and task requirements.

• Advantages Over Existing Solutions:

  1. Shifted the focus from technical/functionality enhancements to "how users adjust usage in complex scenarios."
  2. Combined technical approaches with social and interactive design methods rooted in real-world work practices.

• Experimental or Evaluation Results: The study interviewed 19 participants from diverse fields and experience levels, effectively revealing different perspectives and challenges in AutoML's real-world usage.

• Limitations and Future Directions:

  1. Sampling limitations: Participants were primarily recruited via corporate mailing lists and social media; future studies should expand sample diversity.
  2. Need to explore emerging challenges, such as fairness and bias issues, and AutoML's adaptability in non-English languages or different cultural contexts.
  3. Proposed solutions should be translated into more quantifiable research to support universally applicable and specific design improvements.

Design Implications

1. Support for Domain Customization:

   • Develop AutoML platforms with greater domain-specific focus and editability.
   • Introduce prior knowledge to guide model generation in more complex scenarios.

2. Multi-dimensional Transparency:

   • Provide tools for dynamic process and result transparency.
   • Design explanatory visualization features suitable for non-expert users.

3. Privacy Risk Prevention:

   • Integrate privacy protection notifications and privacy-enhancing technologies.
   • Encourage users to minimize and anonymize data before uploading.

4. Support for Cross-Team Collaboration:

   • Optimize AutoML platform adaptability by incorporating roles from different organizational teams (data scientists, legal teams, domain experts).
   • Introduce design principles to support "Humans-in-the-Loops" (multi-person collaborative loops).

Acknowledgments and Research Support

This research was funded by the National Science Foundation (NSF) under project numbers 2212323, 1951729, and 1953893.