Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation

Title of the Paper

Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation

Paper Information

• Domain: Human-Computer Interaction, Explainable Artificial Intelligence, Collective Creativity
• Keywords: Explainable Artificial Intelligence, Collective Creativity, Diversity, Crowdsourcing, Creativity Support Tools

Research Background and Problem

• Problem and Challenges: Existing creativity support tools rely on human evaluation feedback (e.g., expert or peer reviews), which is difficult to scale in large crowds and cannot provide real-time feedback. Additionally, many tools focus on improving creativity quality while neglecting the enhancement of creativity diversity.
• Significance: In collective creativity scenarios, avoiding repetitive ideas and enhancing overall diversity is particularly important, as society values H-type creativity (truly novel ideas) more than P-type creativity (personal creativity).
• Research Motivation: There is a need for a scalable and real-time automated feedback method to improve both creativity quality and diversity. Furthermore, an explainable feedback mechanism is required to help users understand why certain ideas receive low scores and how to improve them.

Solution

• Method and Solution: The authors propose a method called "Interpretable Directed Diversity," which leverages Explainable Artificial Intelligence (XAI) to predict creativity quality and diversity scores and generate multidimensional explainable feedback, including Attribution, Contrastive Attribution, and Counterfactual Suggestions.
• Innovations:
  1. Provides automated multidimensional feedback, including score prediction, issue highlighting, comparative analysis, and alternative term suggestions.
  2. Supports multiple iterations, allowing users to gradually improve their ideas based on feedback, enhancing overall quality and diversity.
  3. Implements an interactive crowdsourcing system to support large-scale real-time creativity.
• Implementation Steps and Key Techniques:
  1. Use machine learning models to predict quality scores and heuristic methods to calculate diversity scores.
  2. Provide explainable feedback:
     • Attribution Explanation: Highlights words in the idea text that significantly negatively impact the score, guiding improvements.
     • Contrastive Attribution Explanation: Compares word changes between iterations and their contributions to score increases or decreases.
     • Counterfactual Suggestions: Generates alternative term suggestions based on the ConceptNet knowledge graph to enhance quality or diversity.
  3. Present feedback in real-time through an interactive system, supporting users in iterative idea improvement.

Research Outcomes

• Specific Results:
  1. Demonstrated that providing attribution and contrastive attribution feedback significantly improves idea diversity.
  2. Diversity improvements were validated through computational metrics (e.g., total MST edge weight and mean pairwise distance) and human evaluations.
  3. Explainable feedback was generally easy to understand and valuable for creativity.
• Comparison with Existing Approaches:
  1. The proposed method is more scalable than traditional methods relying on expert or peer evaluations.
  2. Provides real-time feedback and multiple explanation types, addressing the issue of incomprehensible scoring.
• Experimental or Evaluation Results:
  1. User experiment results showed that approaches incorporating explainable feedback were most effective in enhancing creativity diversity, especially contrastive attribution feedback (SAX).
  2. The study found that counterfactual suggestions were less effective in improving creativity quality, potentially due to issues with grammar and semantic relevance.
  3. Validation studies indicated that users favored quality scoring, but most feedback types effectively improved computational metrics for creativity quality and diversity.
• Limitations and Future Directions:
  1. Limitations: Counterfactual suggestions sometimes provided irrelevant terms that could disrupt the creative process. Quality score explanations lacked semantic depth, potentially limiting users' understanding of quality.
  2. Future Directions: Improve the grammar and relevance of counterfactual suggestions; explore more advanced explanation types; extend the method to other creative domains, such as story writing and graphic design; develop domain-specific explainable models for complex tasks.

By proposing an AI-driven approach that integrates multiple explainable feedback techniques, the paper demonstrates the potential of the method in supporting collective creativity and lays the foundation for future extensions to other domains and tasks.