Hardhats and Bungaloos: Comparing Crowdsourced Design Feedback with Peer Design Feedback in the Classroom

Title of the Paper

"Hardhats and Bungaloos: Comparing Crowdsourced Design Feedback with Peer Design Feedback in the Classroom"

Paper Information

• Subject Area: Design feedback mechanisms in HCI education
• Keywords: crowdsourcing, design feedback, peer review, HCI, classroom study, UX design, feedback quality, Amazon Mechanical Turk, learning assessment, user experience

Research Background and Problem

• Problems and Challenges:

  • Feedback in design is crucial for education, yet there is limited in-depth research on how students perceive crowdsourced design feedback compared to peer feedback.
  • While crowdsourcing feedback mechanisms offer the potential for scalable feedback collection, their quality, effectiveness, and fairness in influencing students' learning experiences remain unclear.
  • A new area of focus is how students estimate the monetary value of crowdsourced feedback, especially when they benefit from it.

• Significance of the Research:

  • Understanding the differences between crowdsourced and peer design feedback can help improve teaching methods and enhance learning outcomes.
  • Crowdsourced feedback, as a boundary-expanding tool, can expose students to diverse perspectives from outside the classroom.
  • Researching the monetary valuation of design feedback can optimize the pricing mechanisms of crowdsourced tasks, particularly when funded by educational institutions.

• Motivation and Related Work:

  • Previous studies have found that crowdsourced feedback models, such as anonymity, can reduce bias and increase the independence of feedback performance.
  • In educational contexts, peer and crowdsourced feedback are considered to have respective advantages and disadvantages, but their specific performance and students' perceptions of these feedback types remain unclear.
  • Existing literature, such as the work by Wauck et al., has explored comparisons between peer feedback and feedback from various online crowds. This study partially extends and replicates those findings.

Solution

• Proposed Research Methodology:

  • Using the online crowdsourcing platform Amazon Mechanical Turk (MTurk) and classroom peer feedback as research subjects, the study compares the perceived quality and fairness of feedback from both sources.
  • Feedback was collected on design prototype applications provided in an undergraduate HCI course.
  • The study analyzes students' experiences and perceptions of the feedback, as well as their evaluation of the monetary value of crowdsourced feedback, while also examining the potential impact of feedback models on learning outcomes.

• Innovative Contributions:

  • Introducing an exploration of the monetary value of crowdsourced design feedback, expanding the traditional research focus on feedback quality analysis.
  • Analyzing the multidimensional factors influencing design feedback, including perceived efficiency, fairness, emotional tone of feedback, and the professional background of feedback providers.

• Implementation Steps and Key Techniques:

  1. Students created prototype designs for mobile applications, which were then reviewed by peers in the course and MTurk workers.
  2. Feedback was evaluated across various dimensions (specificity, actionability, interpretability, etc.) using surveys and quantitative assessment methods.
  3. A mixed-methods approach was employed to analyze qualitative and quantitative data.
  4. Significant differences between the two feedback sources were validated, and students' subjective evaluations of the monetary value of crowdsourced feedback were explored.

Research Findings

• Specific Findings:

  1. Students generally rated peer feedback higher than crowdsourced feedback in terms of specificity, actionability, interpretability, and overall quality.
  2. The diversity of crowdsourced feedback (especially from individuals with different cultural backgrounds) was considered a unique advantage by some students, with approximately 25% preferring crowdsourced feedback.
  3. Most students underestimated the motivations and actual work quality of MTurk workers while overestimating their professional expertise.

• Comparative Advantages Over Existing Solutions:

  • This study further validates the feasibility of crowdsourced feedback in design education and clarifies its strengths and limitations in teaching contexts.
  • Quantitative research on students' subjective perceptions of the monetary value of feedback complements traditional studies on feedback mechanisms.

• Experimental or Evaluation Results:

  • On a Likert scale, peer feedback scored significantly higher than crowdsourced feedback across multiple dimensions, including process efficiency, fairness, and satisfaction.
  • Except for the emotional tone (valence) of the feedback, all other metrics showed statistically significant differences (p < 0.001).
  • Crowdsourced workers invested less time and effort compared to peers, and some feedback was deemed "meaningless" or "perfunctory."

• Limitations and Future Directions:

  • The study did not control for students' prior knowledge of the crowdsourcing model, which may have influenced their behavior and feedback evaluations.
  • A high proportion of "meaningless" responses in crowdsourced feedback suggests the need for future research to address this issue.
  • The study proposes combining online crowdsourcing with peer feedback and developing support tools (e.g., feedback aggregation and visualization tools) to enhance students' user experience and perceived effectiveness.