From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model

Paper Title

From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model

Publication Info

• Topic area: AI-assisted design feedback and mentorship
• Keywords: Large Language Models, Cognitive Apprenticeship Model, design feedback, visualization, scaffolding, mentoring, structured prompting, reflective practice, user study, AI-human collaboration

Background and Problem

• Problem / challenge: Current AI chatbots provide generic, one-size-fits-all design feedback that limits reflective engagement and fails to foster deeper design reasoning.
• Significance: Effective design feedback is essential for improving artifacts and fostering cognitive development in practitioners, but human mentorship is often unavailable due to resource constraints.
• Motivation and related work: Prior research has demonstrated the potential of LLMs for design critique but highlighted their transactional nature and lack of interactive, reasoning-driven feedback. Human mentors offer dynamic feedback practices that align with the Cognitive Apprenticeship Model (CAM), which this paper seeks to operationalize for AI systems.

Solution

• Proposed approach: DesignMentor, a cognitively-informed AI feedback system that applies CAM principles through structured prompting to transform LLMs into process-oriented design mentors.
• Novelty:
  1. Development of a design mentorship codebook based on CAM and human expert practices.
  2. Implementation of seven design guidelines to operationalize CAM principles in AI feedback.
  3. Empirical evaluation demonstrating DesignMentor’s effectiveness in fostering design reasoning and reflective practice.
• Procedure and key techniques:
  • DesignMentor uses structured prompts to guide users through three feedback phases: articulating goals, diagnosing designs, and reflecting on solutions.
  • Prompts incorporate CAM methods such as modeling, coaching, scaffolding, articulation, reflection, and exploration.
  • Iterative refinement of prompts based on pilot testing with visualization practitioners.
  • Evaluation through a controlled user study comparing DesignMentor to a baseline chatbot.

Results

• Concrete findings:
  • DesignMentor significantly increased user engagement, reasoning articulation, and interaction dynamics compared to the baseline.
  • Feedback spanned broader levels of visualization design, including domain problem characterization and task abstraction.
  • Participants rated DesignMentor higher in feedback completeness, metacognition, motivation, and delivery quality.
• Advantage over baselines:
  • DesignMentor facilitated deeper reasoning and provided more structured, contextually relevant feedback.
  • Users preferred DesignMentor in exploratory and developmental design phases, while the baseline was favored for evaluative tasks.
• Experiments / evaluation:
  • Mixed-methods study with 24 visualization practitioners using their own artifacts.
  • Quantitative measures included feedback completeness, conversational dynamics, and user ratings.
  • Qualitative analysis examined user preferences and feedback characteristics.
• Limitations and future work:
  • Lack of multimodal feedback (e.g., visual examples).
  • Short-term study duration; future work should explore longitudinal deployments.
  • Generalizability to other design disciplines beyond visualization.

Summary

This paper introduces DesignMentor, an AI system that transforms LLMs into design mentors by applying the Cognitive Apprenticeship Model through structured prompting. DesignMentor fosters reflective design reasoning and provides contextually tailored feedback, outperforming baseline chatbots in user engagement and feedback quality. While particularly effective in exploratory and developmental phases, its structured approach requires greater cognitive effort from users. Future research should focus on adaptive systems, multimodal feedback integration, and broader applicability across design domains.