Explaining It Your Way - Findings from a Co-Creative Design Workshop on Designing XAI Applications with AI End-Users from the Public Sector

Document Title

Explaining It Your Way - Findings from a Co-Creative Design Workshop on Designing XAI Applications with AI End-Users from the Public Sector

Document Information

• Thematic Area: Human-Computer Interaction (HCI), Explainable Artificial Intelligence (XAI), AI Application Design in the Public Sector
• Keywords: Human-Centered AI, Explainable Artificial Intelligence, User-Centered Design, Co-Creation, Focus Groups, Social Evaluation, Unemployment Insurance

Research Background and Issues

• Problems and Challenges:

  1. The complexity and opacity of AI systems make it difficult for public sector users to understand and trust AI-based decision-making systems.
  2. Current AI system designs in the public sector lack a user-centered perspective, particularly for end-users (e.g., unemployment counselors).
  3. Few studies have directly involved professional users from the public sector in the design process of explainable AI systems.

• Significance:
  In the field of social services, AI decisions have a significant impact on individuals' lives. Transparent and explainable AI design not only helps improve user acceptance and trust but also ensures fairness and ethical compliance.

• Research Motivation and Related Work:

  1. The goal of Explainable Artificial Intelligence (XAI) is to provide transparent decision explanations to address the "black box" problem in deep learning models.
  2. Current XAI research in the public sector primarily focuses on applying existing technologies (e.g., LIME or SHAP) rather than developing new, need-specific approaches.
  3. Involving end-users in the design process helps improve the practicality and societal impact of AI applications.

Solution

• Methodology or Solution:
  The authors employed a co-creative workshop approach, collaborating with unemployment counselors from the Estonian public sector to design AI user interfaces.

  1. By combining user-centered design processes with problem-driven XAI design principles, the workshop guided participants step-by-step from identifying user needs and pain points to designing interface prototypes.
  2. Using the AI tool OTT from Estonia's social services, the study explored how interpretive interfaces could enhance user experience.

• Innovations:

  1. This study is the first to directly involve professional users from the public sector in the design process of explainable AI interfaces.
  2. It proposed using text-based rather than graphical explanations as the primary interaction method to reduce cognitive load.
  3. It provided a practical design framework and specific recommendations to support the long-term development of AI applications in public sector services.

• Implementation Steps:

  1. Persona Definition and User Journey Mapping: Create user personas for AI users and analyze their workflows.
  2. Pain Point Synthesis and Validation: Identify and categorize issues related to the OTT system (e.g., data gaps, interface complexity, lack of system empathy).
  3. Prototype Design: Design paper-based interface prototypes to address pain points and iterate based on user feedback.
  4. Focus Group Discussions: Review the workshop with participants and gather in-depth feedback on OTT acceptance and improvement suggestions.

Research Outcomes

• Specific Outcomes:

  1. Designed two paper prototypes to improve the interpretability and user acceptance of the OTT system.
  2. Identified three categories of user needs to be addressed:
     • Providing explanations for prediction reasons
     • Offering improvement suggestions
     • Delivering global and local transparency training courses
  3. Highlighted that text-based interpretive interfaces are more suitable for the busy workflows of unemployment counselors than graphical interfaces.

• Advantages:

  1. The co-creation approach increased user acceptance and understanding of OTT, fostering trust in AI tools.
  2. Results demonstrated that directly involving users in the design process can uncover previously unrecognized issues and lead to practical solutions.

• Experimental or Evaluation Results:

  1. Participant feedback indicated that the new designs significantly reduced the risk of misunderstanding OTT's prediction results.
  2. User experience improved, particularly in terms of comprehensibility and usability.

• Limitations and Future Directions:

  1. Limitations:
     • Due to time constraints, the study focused on addressing only one pain point.
     • Participants provided limited background knowledge, which may have constrained the design process.
  2. Future Directions:
     • Advocate for increased involvement of users as primary designers.
     • Explore how participant feedback can be directly integrated into AI models.
     • Extend the applicability of the co-creation approach to other public sector use cases.

Recommendations

1. In designing AI for the public sector, focus on the needs of service recipients, not just service users, to enhance societal benefits.
2. Consider specific cultural and ethical contexts when designing AI systems to align with local user acceptance.
3. Engage with original developers during the early stages of system design to clarify constraints and enable more effective redesigns.
4. Improve the quality of user education programs, including training on AI fairness, ethics, and principles.