Comparing Apples and Oranges: Human and Computer Clustered Affinity Diagrams Under the Microscope

Document Title

Comparing Apples and Oranges: Human and Computer Clustered Affinity Diagrams Under the Microscope

Document Information

• Domain: Human-Computer Interaction and User Research
• Keywords: Text mining, affinity diagrams, automation, design research, clustering, user-centered design, fastText, human-computer collaboration, natural language processing, design efficiency

Research Background and Problem

• Problems and Challenges:

  • Affinity diagrams are essential tools in user-centered design for analyzing and organizing user statements or observations. However, manually creating affinity diagrams is time-consuming and often subjective due to differences among design team members.
  • With advancements in text mining and neural networks for processing qualitative data, researchers aim to explore how algorithms can support the creation of affinity diagrams. However, effectively evaluating the quality of algorithm-generated affinity diagrams remains a challenge.

• Research Importance:

  • Affinity diagrams help design teams extract key insights from user data, supporting the generation of design inspiration. Discovering efficient and accurate methods for creating affinity diagrams would bring significant value to practice.
  • The rapid development of AI technologies offers potential technical support for optimizing affinity diagram generation, which is crucial for reducing time costs and improving user research and design efficiency.

• Research Motivation:

  • To explore how automation technologies, particularly text mining models, can enhance the user research process.
  • To evaluate the effectiveness of algorithm-generated affinity diagrams through quantitative and qualitative methods, identify limitations, and propose directions for improvement.

Solution

• Methods or Solutions:

  • This study compares seven text mining models (e.g., fastText, word2vec, LDA) and selects fastText as the optimal model for generating affinity diagrams.
  • In the experiment, a comparative study was designed using pre-clustered and randomly ordered user data to generate affinity diagrams, exploring the support provided by fastText-generated clusters to design teams.
  • Multiple metrics (technical, psychological, and performance-related) were used to quantify algorithm performance, combined with qualitative feedback from design teams and experts to analyze its impact on the design process.

• Innovations:

  • Systematic comparison of language model-based clustering (e.g., fastText) with human-generated affinity diagrams, and the proposal of evaluation metrics tailored for affinity diagrams.
  • Insights into why current automated affinity diagrams fail to effectively assist design teams, providing a basis for improving algorithms and design support tools.

• Implementation Steps and Key Technologies:

  1. Model Comparison: Evaluate the performance of seven different text mining models, including traditional frequency vector models and context prediction models.
  2. Experiment Design: Invite design teams to construct affinity diagrams using randomly ordered and fastText pre-clustered data, studying differences in efficiency and quality.
  3. Expert Evaluation: Experienced design experts provide subjective quality ratings for both automated and human-generated affinity diagrams.
  4. Qualitative Analysis: Collect feedback from design teams and experts on the use of automated affinity diagrams, analyzing algorithm effectiveness and associated issues.

Research Results

• Specific Results:

  1. Among the seven text mining models evaluated, fastText performed best in semantic similarity tasks (Spearman’s ρ = 0.4431) and was selected as the primary tool for generating affinity diagrams.
  2. The average overlap index between automated affinity diagrams generated by fastText and human-generated diagrams was 0.694 (SD = 0.034), but the Jaccard index between clusters was relatively low (M = 0.30), indicating significant differences.
  3. In student team experiments, pre-clustered data did not improve efficiency; instead, it led to more discussions about technical feasibility and skepticism toward the algorithm.

• Advantages and Limitations:

  • Advantages: Technically, fastText can process large amounts of data quickly, and its subword handling capability performs well in cases of high linguistic complexity.
  • Limitations:
    • FastText clustering is primarily based on keywords, lacking deep semantic understanding, which prevents the generated affinity diagrams from fully meeting designers' needs for key insights.
    • Design teams' lack of understanding of automated technology and distrust in clustering results limits practical application.

• Experiment and Evaluation Results:

  1. Design teams reported low satisfaction with pre-clustered data, showing no significant improvement in subjective workload or task completion progress.
  2. Experts rated fastText-generated affinity diagrams poorly, citing a lack of intrinsic semantic consistency and inability to directly support design insights.
  3. Qualitative interviews revealed that both design teams and experts believe current technical support requires greater transparency and customization improvements.

• Future Directions:

  • Develop semi-automated support tools that integrate designer interaction rather than fully replacing manual operations with automation.
  • Achieve higher levels of semantic understanding (e.g., capturing user statements' emotions and context) to enhance clustering depth and consistency.
  • Further explore algorithm parameter optimization and context-based clustering recommendation systems tailored to different design scenarios.

Conclusion: This study finds that existing text mining technologies have limited practical effectiveness in generating affinity diagrams, but their potential can be realized through targeted follow-up research and tool design improvements.