From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible ML

Title of the Paper

From Plane Crashes to Algorithmic Harm: Applicability of Safety Engineering Frameworks for Responsible ML

Paper Information

• Field of Study: Ethics and Social Responsibility in Machine Learning
• Keywords: Empirical Study, Safety Engineering, Machine Learning, Social and Ethical Risks

Research Background and Problem Statement

• Identified Issues or Challenges:
  • Current processes for evaluating and mitigating social and ethical risks in machine learning (ML) systems are fragmented and lack consistency.
  • Both academia and industry lack a unified understanding of the definition and management methods for social and ethical risks in ML systems.
  • Practical challenges include vague definitions, time and resource constraints, conflicts in organizational culture, and motivational mechanisms.
• Significance:
  • ML systems can cause significant negative impacts, such as inequitable allocation of social resources, reinforcement of societal biases, and exacerbation of inequality. These social and ethical risks need to be systematically identified, assessed, and mitigated.
  • Developing frameworks to address these issues can support international policy and standards development while promoting ethicality and safety in ML systems.
• Research Motivation:
  • The authors aim to explore whether traditional safety engineering frameworks (Failure Mode and Effects Analysis, FMEA, and System Theoretic Process Analysis, STPA) can be applied to manage social and ethical risks in ML systems, addressing structural gaps in current practices.
  • Given the lack of empirical studies on risk management frameworks for ML systems in existing literature, the authors seek to better understand the applicability and potential improvements of these frameworks through interviews with industry practitioners.

Solution

• Proposed Methods or Solutions:
  • Employ analytical methods from FMEA and STPA safety engineering frameworks to improve practices for assessing social and ethical risks in the ML domain.
  • Conduct semi-structured interviews with industry practitioners to gather firsthand data and evaluate the applicability and practical value of these frameworks.
• Innovative Aspects:
  • Extend safety engineering methods traditionally applied in high-risk domains like aviation and nuclear energy to the social and ethical risks of ML systems for the first time.
  • Provide a comprehensive comparative perspective, analyzing the complementary aspects of the two frameworks (FMEA and STPA) in managing social risks in ML.
• Implementation Steps:
  1. FMEA Process: Break down the functions or steps of the ML system, identify potential failure modes, analyze their impact, causes, and possible detection methods.
  2. STPA Process: Establish a system-level control structure, define potential losses, hazards, and unsafe control actions, and derive failure scenarios.
  3. Through interviews, the authors guided participants to use these frameworks to simulate analyses of social risks in ML systems.

Research Findings

• Specific Findings:
  • Advantages of FMEA: Provides tools for breaking down specific ML system functions and assessing failure modes, helping to clarify technical functions, potential risks, and responsibilities.
  • Advantages of STPA: Reveals potential harms in interactions between ML systems and society or stakeholders through a perspective of system interactions and feedback loops.
  • Identified the complementary nature of the two frameworks: FMEA is better suited for analyzing specific modules or processes, while STPA is more effective for examining complex system interactions.
• Advantages Over Existing Solutions:
  • Compared to existing non-systematic methods for social risk assessment, these safety engineering frameworks offer systematic and repeatable methodologies.
  • By providing clear steps and processes, they alleviate challenges in multidisciplinary collaboration caused by inconsistent terminology across fields.
• Experimental or Evaluation Results:
  1. Feedback on FMEA Application: Encountered difficulties during the functional breakdown phase, especially for large ML models; additionally, there was uncertainty in quantifying the severity and likelihood of social risks.
  2. Feedback on STPA Application: Defining system boundaries and representing complexity were major challenges when involving multiple stakeholders and technical interactions; participants generally felt the STPA framework required clearer definitions and analyses of social contexts.
• Limitations and Future Directions:
  • Limitations:
    • FMEA and STPA applications often assume a thorough understanding of the system deployment environment, which is frequently exceeded by the complexity of ML systems.
    • The lack of long-term organizational transformation support mechanisms hinders the practical feasibility of these frameworks in industrial settings.
  • Future Research Directions:
    • Optimize existing frameworks to better adapt to the social and ethical risks of ML, particularly in modeling interactions with social contexts and stakeholders.
    • Incorporate perspectives from critical theory and feminism to address deficiencies in assessing social diversity and equity within current frameworks.
    • Conduct in-depth studies on the practical needs and adaptation strategies for implementing these frameworks across different industries and company sizes.

Conclusion

This study provides an initial empirical foundation for introducing safety engineering frameworks (FMEA and STPA) into the management of social and ethical risks in ML systems. Despite theoretical and practical challenges, the structured methods offered by these frameworks open new possibilities for improving the fragmented and unsystematic risk management practices currently in place. Additionally, the paper identifies potential directions for further theoretical refinement and industrial application of these frameworks, serving as a guide for future work.