"Should I choose a smaller model?'': Understanding ML Model Selection and Its Impact on Sustainability

Research Background and Issues

• Identified Problems and Challenges:
  The paper focuses on sustainability issues during the machine learning (ML) model selection phase, particularly in cases where models have significant environmental impacts due to energy and resource consumption. The authors observed that, despite the growing attention to sustainability topics in current ML and human-computer interaction (HCI) research, developers rarely consider sustainability as a key factor when choosing ML models. The rise of large-scale, multipurpose models (e.g., ChatGPT) exacerbates this issue, as these models often deliver high performance but consume substantial computational and energy resources.

• Significance:
  Current ML development and deployment have already had a significant impact on global energy consumption and carbon emissions. As model scale and complexity continue to increase, this impact is expected to worsen. For example, it is estimated that energy consumption related to AI and data centers alone could reach levels comparable to the energy demands of an entire country like Japan in the coming years. Prioritizing and evaluating sustainability during the model selection phase is critical to addressing this issue.

• Research Motivation and Related Work:
  While existing research has proposed various algorithms and tools to optimize model energy consumption, most studies focus on the training phase, with limited consideration for energy consumption during inference and the sustainability of subsequent ML lifecycle stages. Additionally, current methods often fail to adequately support developers in balancing performance, interpretability, and sustainability during model selection.

Solution

• Proposed Methods or Solutions:
  Through semi-structured interviews with 13 ML developers, the authors conducted an in-depth analysis of the factors considered by developers during model selection, their awareness of sustainability, and the characteristics of their trade-off processes. The study also provides recommendations to support sustainable practices.

• Innovative Contributions:
  This research is the first to systematically reveal the practical workflows, major challenges, and cognitive blind spots in model selection from the developers' perspective, particularly regarding the lack of awareness about sustainability and the trade-offs between model complexity and environmental impact. This user-centered research approach is unprecedented in the existing technical literature.

• Implementation Steps and Key Techniques:
  The authors employed the following methods:

  1. Interview Design and Data Collection: Designed detailed semi-structured interviews focusing on developers’ model selection processes, parameter prioritization, trade-off factors, and understanding of sustainability.
  2. Data Analysis: Used a mixed thematic analysis approach to code and summarize the data collected from the interviews.
  3. Pattern Identification: Extracted key trends in model selection behaviors, decision-making processes, and sustainability considerations, and translated the findings into actionable recommendations to support developers' decision-making.

Research Findings

• Specific Findings:

  1. Developers’ understanding of datasets, application scenarios, and end-user needs forms the critical foundation for model selection, but there is almost no systematic approach to considering sustainability issues.
  2. Younger developers tend to prefer complex, large pre-trained models (e.g., GPT) because these models are emphasized during their education and are easily accessible, yet they have limited awareness of the environmental impact of such models.
  3. Almost all interviewees did not consider energy consumption or infrastructure costs as important metrics in model selection (e.g., rarely tracking model runtime energy consumption or considering the sustainability of cluster hardware).

• Comparison with Existing Solutions and Advantages:
  Compared to algorithm-centric studies, this research focuses on exposing cognitive blind spots and practical challenges in developers' decision-making processes. It provides actionable guidance for building human-computer collaboration tools and raising developers' awareness of sustainability. Additionally, the study offers recommendations for educational and technical tools to help developers form a more comprehensive consideration framework during model selection.

• Experimental or Evaluation Results:

  • Most developers only consider sustainability after project completion, and only three developers had used tools to track model carbon footprints.
  • While developers generally prioritize performance metrics (e.g., accuracy), trade-offs regarding interpretability, fast inference time, and GPU energy usage largely rely on personal experience and lack systematic approaches.

• Limitations and Future Directions:

  • Limitations: The study’s sample size (13 developers) is relatively small, with participants primarily from research institutions or personal networks, which may introduce selection bias. Additionally, the proportion of female participants was low, reflecting the gender distribution in the ML industry.
  • Future Directions: Future research should focus on larger samples with developers from diverse professional backgrounds, especially early-career developers or end-users of ML models. Moreover, sustainability research should expand to include social and economic impacts (e.g., fairness and accessibility) rather than focusing solely on environmental impacts.

Conclusion

The authors clearly describe the pain points and opportunities in current ML developers’ model selection processes, uncovering bottlenecks in balancing performance, interpretability, and sustainability. The study suggests incorporating sustainability into ML education and adopting broader research methodologies to encourage developers to consciously implement environmentally friendly practices and model selection strategies during system design and deployment. This has practical significance for building a more responsible AI ecosystem within the ML and HCI communities.