Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Support

Title of the Paper

Improving Human-AI Collaboration in Child Welfare: Understanding Practitioners' Practices, Challenges, and Needs for Algorithmic Decision Support

Bibliographic Information

• Subject Area: Application of AI-assisted decision-making in high-risk social welfare work
• Keywords: Algorithmic decision support, human-AI collaboration, child welfare, artificial intelligence, social work, decision support tools, contextual analysis, trust and reliance, algorithmic fairness, organizational culture

Research Background and Problem Statement

• Problems and Challenges:

  • Algorithmic Decision Support Systems (ADS) have been introduced into high-risk social work domains, including child welfare. These tools promise to enhance efficiency and fairness by supplementing or improving human decision-making but may also introduce biases or other forms of error.
  • A typical case of ADS usage, the "Allegheny Family Screening Tool (AFST)," has been deployed for several years. However, it remains unclear how frontline workers perceive and use this tool in practice, as well as the primary challenges and needs they face.
  • There is a lack of comprehensive understanding of how humans and AI systems complement each other in real-world environments, particularly in the context of social work within the public sector.

• Significance of the Research:

  • Decision-making processes are critical to child safety and the future of their families. In the high-risk domain of child protection, ensuring the reliability of algorithmic tools and their effective integration with human decision-making processes is essential to avoid irreparable harm to families and individuals.
  • A deeper understanding of the role of algorithms in practice can inform the design of more equitable and efficient human-AI collaboration models.

• Motivation and Related Work:

  • Existing research primarily focuses on retrospective quantitative analyses, with limited attention to the perspectives of frontline workers in real-world usage scenarios.
  • Previous studies have highlighted the potential of human-AI collaboration to improve decision-making outcomes but have also found that inefficiencies, lack of transparency, or misalignment with human goals can undermine user confidence.
  • This study builds on existing theories of social decision-making and human-computer interaction design frameworks to identify methods for fostering more effective human-AI partnerships.

Proposed Solution

• Research Methods:

  • Conducted fieldwork at a child welfare agency in Allegheny County, Pennsylvania, including contextual inquiry and semi-structured interviews.
  • Collected data through 37.5 hours of on-site observation and interviews with 9 call screeners and 4 supervisors to understand their current practices, challenges, and potential design improvements for using AFST.

• Innovation and Methodology:

  • Provides the first in-depth qualitative study of AFST usage scenarios, addressing gaps in prior research that focused on quantitative data.
  • Explores inconsistencies between ADS and human decision-making goals based on the specific experiences of social workers and their implications.

• Implementation Steps and Techniques:

  • Used contextual inquiry methods to observe workers' behaviors during call screening, data entry, and the use of AFST scores.
  • Validated observations through interviews to gain deeper insights into workers' perceptions, experiences, and suggestions for improvement.

Research Findings

• Key Findings:

  • Workers compensate for the shortcomings of algorithmic tools by leveraging their own expertise to calibrate their reliance on algorithmic outputs. For example, they consider contextual details (e.g., reporter intent, cultural misunderstandings) that are often not captured by the algorithm.
  • Workers primarily learn about AFST through informal means, such as discussions with colleagues and observing score patterns. However, this self-acquired knowledge may be incomplete or even misleading.
  • Organizational pressures and incentives, such as implicit rules to "avoid overriding algorithmic recommendations too often," significantly influence workers' decision-making.
  • The long-term risk prediction focus of AFST (e.g., risk of re-placement within two years) does not align with workers' short-term safety priorities. This misalignment is a major source of friction in using the tool.

• Strengths:

  • The study highlights how workers use human-specific contextual knowledge to address blind spots in algorithmic models, emphasizing the potential for human-AI complementarity in ADS design.
  • It underscores the impact of institutional factors, such as organizational culture and workflows, on frontline workers' trust in and reliance on algorithms, enriching the theoretical foundation for applying algorithmic decision support in public service contexts.

• Experimental or Evaluation Results:

  • While some workers recognized the potential benefits of the algorithm, a lack of understanding of model transparency, prediction goals, and score generation logic led to widespread skepticism.
  • Interviews and observations revealed that many workers view the algorithm as an "auxiliary tool" rather than a definitive decision-making authority.

• Limitations and Future Directions:

  • Limitations:
    • The study focuses on a single case (AFST), and the generalizability of its findings needs further validation in other social work contexts.
    • The small sample size may not fully represent the diverse experiences of a broader group of workers.
  • Future Directions:
    • Enhance the transparency of algorithmic tools and provide better training for workers to enable more effective tool usage in decision-making.
    • Explore how stakeholder participation during the design phase can reduce inconsistencies between human decision-making goals and algorithmic prediction objectives.
    • Conduct more systematic research on the long-term impact of organizational culture and incentive structures on trust in algorithms.