Impressions and Strategies of Academic Advisors When Using a Grade Prediction Tool During Term Planning

Document Title

Impressions and Strategies of Academic Advisors When Using a Grade Prediction Tool During Term Planning

Document Information

• Subject Area: Academic advising and data-driven course recommendation tools
• Keywords: Academic advising, course recommendation, student grade prediction, academic data analysis, human-computer interaction, GPA impact, learning analytics dashboard, data-driven decision-making

Research Background and Issues

• Identified Problems or Challenges:

  1. Academic advising must consider students' individual circumstances and course structures. However, limited advising time and a large number of students increase workload pressure.
  2. While many course recommendation tools exist for students, there is limited research on how these tools impact advisors' usage and decision-making.
  3. Existing systems struggle to address subjective factors requiring advisors' judgment, such as students' non-academic issues.

• Significance: Academic advising significantly enhances higher education institutions' mission by promoting student academic success, reducing dropout rates, and optimizing academic choices. It has become a mandated practice by educational accreditation bodies.

• Research Motivation and Related Work:

  • Numerous tools have been developed to optimize students' course selection, particularly those leveraging data and AI-based prediction tools. However, how these tools influence advisors' decision-making strategies remains unknown.
  • Previous studies have found that students are easily influenced by grade prediction features in course recommendation tools, often focusing on GPA maximization while neglecting other factors such as workload and time demands.

Solution

• Methods or Solutions:

  1. Developed a grade prediction course recommendation tool named iCoRA, which integrates machine learning models to predict students' GPA ranges for specific courses.
  2. Provided interpretability features to enhance tool transparency by showing the impact of various factors on grade predictions.

• Innovations:

  • Optimized conversations and interactions during advising sessions by supporting course recommendations through a data-driven approach.
  • Included visual representations of course structures, enabling advisors to quickly understand the logical relationships between courses.

• Implementation Steps and Key Technologies:

  1. The tool utilized quantile regression and gradient boosting tree techniques to generate predicted grade ranges (low, average, high).
  2. Offered a "Why" button for interpretability, revealing the contributions of different input variables to grade predictions.
  3. Designed an experiment simulating student profiles with varying GPAs, allowing advisors to recommend courses and evaluate the outcomes.

Research Findings

• Specific Findings:

  • Advisors' usual course recommendation strategies (based on personal experience and teaching knowledge) did not change significantly after using the tool.
  • The tool primarily served as a reference for decision-making, especially for high-GPA students, where recommendations were more challenging. For low-GPA students, recommendations tended to reduce course load and aim for GPA improvement.
  • Advisors spent more time adjusting recommendation strategies for low-performing students.

• Advantages Compared to Existing Solutions:

  • The tool provided data-driven evidence to support course recommendations, introducing greater objectivity.
  • Enhanced the depth of discussions and data transparency during the advising process regarding course selection.

• Experimental or Evaluation Results:

  • Low-performing students showed significant GPA improvement, consistent with advisors' more conservative and optimized recommendation strategies for this group.
  • Historical workload data did not significantly influence advisors' decisions, indicating that advisors relied more on their teaching experience with the courses.

• Limitations and Future Directions:

  1. Challenges remain in tool validation and prediction credibility, as some advisors expressed low trust in prediction results.
  2. The study was limited in sample size and scope, focusing only on computer science program advisors in an engineering school context.
  3. Students did not directly participate, and the synthetic student profiles generated for the experiment may not fully reflect real-world scenarios.
  4. Future work could explore more comprehensive academic assessment metrics, such as non-graded tasks, mastery learning, and data fairness.

Contributions and Significance

This study reveals how tools influence advisors' behavioral strategies during academic advising and proposes new design ideas for human-computer collaboration in addressing educational challenges. It also provides insights into the design and application of predictive data and educational technology tools.