Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Document Title

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Document Information

• Domain: Human-Computer Interaction (HCI), Information Design, Collaborative Design Feedback
• Keywords: Crowdsourced design feedback, feedback framework, sentiment analysis, reflection, questioning techniques

Research Background and Problem

• Identified Issues or Challenges: Traditional design feedback is often presented in declarative form, and the content may carry strong positive or negative sentiment, which affects designers' acceptance of the feedback and the quality of design iterations. Excessively negative or positive feedback can lead to ineffective feedback, emotional fluctuations in designers, and hinder task completion.
• Significance: Effective design feedback can promote productive reflection, generate useful ideas for design improvement, and enhance designers' acceptance of feedback. Crowdsourced feedback systems offer the potential to quickly gather a large volume of feedback, but improving their effectiveness remains a challenge.
• Research Motivation and Related Work: Previous studies have shown that integrating reflection tasks with feedback can improve the quality of design iterations. On the other hand, introducing question-based feedback may neutralize sentiment polarity and improve feedback acceptance, but empirical studies are lacking. The authors aim to verify whether combining question-based feedback with declarative feedback can significantly enhance the quality of design iterations.

Solution

• Proposed Method: Enhance the effectiveness of design feedback by combining open-ended questions with traditional declarative feedback. A "question + statement" feedback framework is proposed, where question-based feedback guides designers' reflection, and declarative feedback provides direct modification suggestions.
• Innovations:
  • First to propose using open-ended questions to guide design reflection combined with declarative suggestions in feedback.
  • Empirically validate that this method improves design iteration quality and mitigates the negative impact of overly emotional feedback.
• Implementation Steps:
  1. Collect declarative feedback from crowdsourced workers.
  2. Require crowdsourced workers to rephrase the statements into open-ended questions, ensuring the questions target the same design aspects and promote reflection.
  3. Test the impact of declarative feedback, question-based feedback, and their combination on design improvement through user experiments.
  4. Evaluate feedback quality and design iteration outcomes using expert reviews and sentiment analysis tools (e.g., VADER).

Research Findings

• Specific Results:
  • First User Study confirmed that non-expert crowdsourced workers can successfully provide high-quality open-ended questions, with 85% of questions meeting the criteria for openness and thought stimulation. Additionally, question-based feedback showed significantly more neutral sentiment polarity compared to declarative feedback.
  • Second User Study demonstrated that combining question-based and declarative feedback (Q+S) significantly outperformed using declarative feedback (S) or question-based feedback (Q) alone. Participants' revised designs based on Q+S feedback received higher quality scores.
• Advantages Compared to Existing Solutions:
  • Combining question-based feedback can neutralize the emotional impact of negative declarative feedback while providing designers with clear and inspiring modification suggestions.
  • Effectively addresses the issue of favoring positive feedback that contributes less to design improvement.
• Experimental or Evaluation Results:
  • Quality scores of design iterations significantly improved under the Q+S condition (effect size d > 0.2).
  • Participants reported that question-based feedback facilitated design reflection, though question-based feedback alone showed weaker performance in design iteration quality.
• Limitations and Future Directions:
  • Did not verify the direct moderating effect of question-based feedback on the perception of negative or positive statements.
  • Research was limited to the graphic design domain; future work could extend to other creative fields.
  • Explore how natural language processing can further enhance the quality of question-based feedback and analyze the thought-stimulating characteristics of specific questions.
  • Investigate how users with varying levels of design experience (professional designers vs. non-professional designers) perceive the impact of question-based feedback.