Interactive Explainable Ranking

Paper Title

Interactive Explainable Ranking

Publication Info

• Topic area: Decision-making tools for explainable ranking across subjective and multifaceted criteria.
• Keywords: Explainable ranking, decision-making tools, cognitive bias, multi-criteria ranking, user insertion sort, AI integration, scalability, ethical guardrails, visualization, optimization.

Background and Problem

• Problem / challenge: Existing decision-making tools (DMTs) focus on either rankings or explanations individually, often assuming fixed criteria and weights. This limits adaptability and fails to address inconsistencies or bias in subjective decision-making tasks.
• Significance: Explainable rankings are crucial for subjective, nuanced decisions such as grading, hiring, or evaluating purchases. Tools that support explainability and adaptability can improve fairness, consistency, and user confidence in high-stakes scenarios.
• Motivation and related work: Previous tools like LineUp, RankASco, and Podium focus on fixed criteria and weights, limiting their applicability to subjective and creative tasks. Cognitive biases such as availability and bounded rationality further complicate decision-making. This paper builds on these limitations to propose a more adaptable and scalable solution.

Solution

• Proposed approach: An interactive decision-making tool that combines visualization, optimization, and optional AI to help users create explainable rankings consistent with their preferences.
• Novelty:
  1. Explanation-Rank Resolution (ERR): Visualizes and resolves conflicts between rankings and explanations.
  2. User Insertion Sort (UIS): Accelerates pairwise comparisons while ensuring human oversight in ranking decisions.
  3. Modular integration of AI and optimization to assist in criteria estimation and weight inference.
  4. Ethical guardrails to mitigate algorithmic bias and prevent misuse.
• Procedure and key techniques:
  • Users independently edit rankings, criteria, and weights.
  • ERR identifies inconsistencies between rankings and explanations, visualized through slope charts and comparison panels.
  • UIS uses pairwise comparisons to refine rankings and integrate AI-estimated insertion orders.
  • AI models like CLIP and GPT-4o assist in estimating criteria values, while SVM optimization infers weights to explain rankings.

Results

• Concrete findings:
  • Consistency improved from 0.63 (baseline) to 0.90 (tool), confidence increased from 4.03 to 5.75, and justification referencing criteria rose from 0.25 to 0.86.
  • Participants defined more criteria using the tool (5.50 vs. 3.13 in baseline; p = 0.009**).
  • Usability score of 85.42 (UMUX-LITE), indicating excellent usability.
• Advantage over baselines:
  • Significant improvements in ranking consistency, user confidence, and explicit justification compared to Google Sheets.
  • Enhanced adaptability and scalability for subjective tasks.
• Experiments / evaluation:
  • Within-subjects study with 8 participants comparing tool vs. baseline on ranking tasks (cities and videos).
  • Two case studies: ranking short films (4 participants) and grading open-ended projects (4 teaching assistants).
  • Metrics included consistency checks, qualitative feedback, and usability surveys.
• Limitations and future work:
  • Challenges in scaling information gathering and balancing subjectivity with quantification.
  • Need for richer logical conditions and composite rules in rankings.
  • Deployment in real-world scenarios and longitudinal studies planned to validate ecological impact.

Summary

This paper introduces an interactive tool for explainable ranking, addressing limitations in adaptability, scalability, and ethical concerns in existing DMTs. Key innovations include Explanation-Rank Resolution and User Insertion Sort, which help users resolve inconsistencies and refine rankings through visualization and pairwise comparisons. Experimental results demonstrate significant improvements in ranking consistency, user confidence, and justification compared to traditional tools. Future work aims to scale the tool to larger datasets, support complex logical conditions, and deploy it in real-world decision-making scenarios.