Seeing Beyond Expert Blind Spots: Online Learning Design for Scale and Quality

Title of the Paper

Seeing Beyond Expert Blind Spots: Online Learning Design for Scale and Quality

Paper Information

• Subject Area: Online learning design in higher education, particularly teaching methods and assessment mechanisms in Human-Computer Interaction (HCI) education.
• Keywords: HCI education, teaching beliefs, learning scalability, learning experience design, multiple-choice questions, matched assessment comparison

Research Background and Issues

• Problems and Challenges:

  • In online learning design, scalability and teaching quality are often seen as conflicting goals.
  • In HCI education, traditional teaching frequently uses open-ended questions to cultivate students' critical thinking, but this approach is inefficient in large-scale teaching. Conversely, multiple-choice questions (MCQs), while easy to automate for grading, are perceived as low-quality and unsuitable for fostering higher-order cognitive skills.
  • The presence of expert blind spots: educators' beliefs may not align with students' actual performance.

• Significance of the Research:

  • Understanding and challenging educators' stereotypes about question design is crucial for improving teaching design in HCI and other fields.
  • Striving to balance large-scale learning and high-quality teaching in online education.

• Motivation and Related Work:

  • Current HCI education research mainly focuses on project-based and case-based learning, emphasizing student interaction with real users, while neglecting scalable designs for early skill training.
  • There is no consensus in academia on whether MCQs can effectively foster higher-order cognitive skills.
  • It is necessary to examine whether educators' preferences for MCQs or open-ended questions align with actual teaching outcomes.

Solution

• Research Methods:

  • Survey Study: Conducted a survey of 22 HCI educators to explore their views on the teaching value of MCQs and open-ended questions.
  • Experimental Study: Designed and tested 18 pairs of matched MCQs and open-ended questions in two university HCI courses to compare student performance.
  • Data Analysis: Used a mixed-effect logistic regression model to analyze the difficulty levels of student responses to both question types and compared educators' predictions with students' actual performance.

• Innovations:

  • Proposed an experimental method based on matched question comparisons to reveal cognitive training characteristics through student performance on different question types.
  • Reassessed the potential of MCQs in fostering higher-order thinking skills, challenging the traditional notion that MCQs only test memory and recognition.

• Key Techniques:

  • Applied mixed-effect models to analyze differences; constructed distractors based on students' past errors to ensure MCQs and open-ended questions targeted the same skills.

Research Findings

• Specific Findings:

  • Student Performance: Data showed no significant difference in student performance between matched MCQs and open-ended questions. This indicates that MCQs are not always "easier" and can sometimes be more challenging.
  • Mismatch Between Educators' Beliefs and Student Performance: Educators generally believed open-ended questions were more difficult, but the experimental results revealed a clear discrepancy between their judgments and students' actual performance.
  • Cognitive Element Analysis: The study found that both MCQs and open-ended questions required evaluation skills, and the act of generating answers in open-ended questions was not the primary difficulty.

• Comparisons and Advantages:

  • Contrasting Traditional Views: This study refutes the common perception that MCQs are low-quality and unsuitable for fostering higher-order skills, providing data to support their large-scale application in HCI education.
  • Complementarity with Complex Project-Based Learning: Introducing MCQs during foundational skill training can prepare students for engaging in complex practical learning in the future.

• Limitations and Future Directions:

  • Limitations: The study focused on specific topics within HCI methods, and the results may not apply to more complex or open-ended learning objectives.
  • Future Directions:
    • Further research into the potential of MCQ design across different fields and learning objectives.
    • Explore optimal ways to combine MCQs and open-ended questions.
    • Promote the establishment of the professional role of Learning Experience (LX) designers to enhance teaching design capabilities in higher education.

Summary and Recommendations

• This study confirms that MCQs can achieve both scalability and high-quality teaching in certain fields, with significant potential for fostering higher-order thinking.
• Advocates for low-cost experiments to test the validity of educational designs, reducing the negative impact of expert blind spots on teaching decisions.
• Recommends introducing dedicated learning experience designers in higher education to achieve the dual goals of scalability and quality in teaching.