"Who is running it?" Towards Equitable AI Deployment in Home Care Work

Research Background and Issues

• Problem or Challenge: The authors focus on the deployment of artificial intelligence (AI) technologies in home care work (HCW) and their impact on home care workers (HCWs). Although AI is considered to have the potential to improve work efficiency, most HCWs currently lack an understanding of how AI systems operate, how data is used, and the potential implications of AI-driven decisions. Additionally, ethical, fairness, and accountability issues associated with AI adoption have not been adequately explored.
• Significance: Home care workers are a critical component of healthcare systems in the United States and many other countries, supporting approximately 48 million people in their daily lives. However, this group is often overlooked as low-wage laborers facing numerous challenges, such as low pay, isolated work environments, and lack of training. The use of AI technologies, if unsupervised, could exacerbate existing inequalities.
• Research Motivation: There is currently little research on the impact of AI deployment on low-wage workers, particularly in the home care industry. Moreover, the voices and needs of HCWs are often excluded from technology development and decision-making processes. The research aims to fill these knowledge gaps and reflect the perspectives of grassroots workers to promote more equitable AI design and governance.

Solutions

• Methods or Solutions: The study employs qualitative research methods, including interviews with 22 participants (comprising HCWs, care agency staff, and labor advocates), to analyze the current and potential future uses of AI in home care work and to discuss the ethical, fairness, and governance issues it raises.
• Innovations:
  • Introduces a multi-stakeholder perspective, focusing on the viewpoints of marginalized groups such as HCWs to explore the potential impacts of AI in sensitive caregiving contexts.
  • Uses vignettes to examine the benefits and risks of AI, as well as stakeholders' trust, perceptions of safety, and governance expectations.
  • Highlights key issues in AI deployment, such as transparency, accountability, and democratic governance.
• Implementation Steps and Techniques:
  1. Interview Design and Execution: Semi-structured interviews were conducted to gather participants' opinions on AI, data usage, and governance.
  2. Qualitative Analysis: A three-stage analysis process (including structured coding) and thematic analysis were used to organize data around research topics and extract key themes.
  3. Case Scenarios: Two fictional scenarios (AI for matching care tasks and sound analysis in patients' homes) were used to guide discussions and collect participants' views on AI's potential functionalities and impacts.

Research Findings

• Specific Findings:

  1. Awareness of the Current Situation:
     • HCWs generally lack basic knowledge about AI and its practical applications in their work. Most were unaware that AI technologies were already being used in their work environments, such as algorithms for shift matching.
     • Care agency administrators demonstrated a higher level of AI awareness, recognizing its efficiency in task allocation, such as shift scheduling.
  2. Risks of AI:
     • HCWs expressed concerns that AI might strip caregiving of its human elements, such as empathy and personalized considerations.
     • The lack of transparency in data collection and retention policies left HCWs confused and worried about how their data might be used or misused.
     • AI could lead to unfair penalties for HCWs due to technical errors or biases, affecting their income and reputation.
  3. Governance and Transparency Expectations:
     • Participants broadly called for new AI regulations and policies to define accountability, ensure fairness, and clarify data management practices.
     • Advocated for democratic governance structures that involve stakeholders, including workers, patients, care agencies, and policymakers, in AI decision-making.

• Advantages Over Existing Work:

  • Focuses on the home care industry, a labor-intensive sector where workers operate in isolated and undervalued conditions.
  • Amplifies the voices of marginalized workers in technology decision-making and governance, which are rarely quantified or analyzed in other AI-related studies.
  • Proposes practical interventions, such as establishing data cooperatives and AI literacy education mechanisms.

• Experimental or Evaluation Results:

  • AI has indeed provided operational efficiency in certain scenarios for care agencies, but there remains a need for policy, regulatory, and educational support.
  • Identified potential threats to fairness, such as discrimination risks stemming from intersecting identities like race and gender.

• Limitations and Future Directions:

  • Limitations:
    1. The sample size is small and limited to a major U.S. city, which may restrict the generalizability of the findings.
    2. The perspectives of other stakeholders, such as technology companies, patients, and policymakers, were not extensively included.
    3. The methodology relied primarily on interviews, which may be subject to recall bias.
  • Future Directions:
    1. Expand the research context to rural areas and global settings outside the U.S.
    2. Conduct field observations or longitudinal studies to explore the real-world impacts of AI-driven tools in greater depth.
    3. Investigate specific AI regulatory and collaborative governance models, such as the applicability of data cooperatives.

Through this research, the authors not only reveal the operations and challenges of existing AI technologies in the home care industry but also provide important recommendations for designing more equitable and human-centered AI systems. These insights offer practical guidance for building cross-disciplinary AI governance frameworks.