Rethinking "Risk" in Algorithmic Systems Through A Computational Narrative Analysis of Casenotes in Child Welfare

Document Title

Rethinking "Risk" in Algorithmic Systems Through A Computational Narrative Analysis of Casenotes in Child-Welfare

Document Information

• Subject Area: Algorithmic Risk Assessment, Social Welfare Systems, Human-Computer Interaction
• Keywords: Computational Narrative Analysis, Risk Prediction, Decision Uncertainty, Risk Work, Social Welfare Practice

Research Background and Problem

• Research Problem: The paper focuses on the concept of "risk" in algorithmic systems within the public sector, particularly in risk assessment for child welfare systems, analyzing the mismatch between system design and social context.
• Significance: Risk assessment algorithms are widely applied in the public sector for decision-making, such as child abuse, unemployment rates, and family housing issues. However, the quantification of risk by these algorithms is often based on limited, biased data, leading to unfair decisions and potentially exacerbating systemic inequities.
• Motivation and Related Work: Current algorithmic systems' understanding of "risk" neglects socio-ecological variables from families and individuals, as well as institutional risk factors. Existing research shows that these algorithms may embed human biases and result in over-surveillance or investigation of vulnerable families, turning systems originally designed to provide support into mechanisms that have adverse effects.

Solution

• Proposed Approach: Utilizing Computational Narrative Analysis, this study analyzes case note texts in child welfare systems to uncover systemic and institutional risks that require attention.
• Innovations:
  • Employing semi-supervised topic modeling techniques (Correlation Explanation, CoREx) with embedded domain knowledge (e.g., anchor words related to risk and protective factors) to extract key themes from the text.
  • Highlighting the temporal dynamics of risk and how institutional factors influence family well-being and street-level decision-making through narratives.
  • Challenging existing algorithms that rely on static constructions of risk, proposing a focus on the ecological and fluctuating nature of risk.
• Implementation Steps:
  1. Collecting child welfare case note texts (over 10,000 records).
  2. Applying data cleaning and preprocessing techniques, such as text anonymization and word standardization.
  3. Conducting topic modeling using the CoREx algorithm to extract key themes and analyzing them with domain knowledge.
  4. Using qualitative axial coding to analyze the relationships and conflicts among risk, protective, institutional, and procedural factors.
  5. Visualizing the temporal changes of the above factors to reveal complex decision-making dynamics.

Research Findings

• Specific Findings:
  • Identified four major influencing factors: risk factors, protective factors, institutional factors, and procedural factors.
  • Revealed that institutional factors and procedural practices themselves could pose risks to families, such as through over-assessment or limiting family autonomy.
  • Found significant interactions and temporal variations between risk factors and protective factors.
  • Through qualitative analysis, uncovered the uncertainty in decision-making and the dynamic nature of case trajectories (e.g., how risk, systemic issues, and procedural constraints intertwine to influence final decisions).
• Advantages:
  • Compared to traditional static, numerical risk assessments, this study emphasizes dynamic and socially contextual analysis, yielding more realistic and empathetic results.
  • Provides new guidelines for designing human-AI collaborative systems, such as improving decision support without compromising family rights or privacy.
• Experimental and Evaluation Results:
  • The model identified 19 key themes, describing multidimensional conflicts and factor variations.
  • In high-complexity cases, decisions exhibited greater uncertainty, with risk, protection, and procedural constraints constantly fluctuating.
• Limitations and Future Directions:
  • The study's sample data is limited to a single child welfare agency in the U.S. Midwest, which may have regional constraints.
  • Data primarily comes from case notes written by social workers, potentially limiting or omitting parents' true preferences and feelings.
  • Future work could expand to data from different regions, broader social welfare domains, and incorporate perspectives from affected families to enrich the analysis.

Summary and Discussion

• Core Conclusions:
  • Risks in child welfare systems stem not only from families but are deeply influenced by institutional and procedural factors.
  • The temporal dynamics and interactions of risk need to be incorporated into algorithmic decision design, but current predictive models overlook these complexities, potentially exacerbating existing inequities.
  • Case notes as a data source have multiple limitations (e.g., biases and insufficient data) and should not be directly used for developing predictive algorithms. More comprehensive socio-ecological models should be adopted when designing AI-assisted decision systems.
• Academic and Practical Significance:
  • This study provides a new dimension to the discussion of "algorithmic fairness" in the child welfare domain.
  • It urges HCI researchers and fairness-focused algorithm developers to consider a "sociotechnical frame," avoiding the oversimplification of social problems into purely technical solutions.
  • Advocates for more human-centered system design approaches, prioritizing the well-being of parents and children as the core objective.