Establishing Heuristics for Improving the Usability of GUI Machine Learning Tools for Novice Users

Title of the Paper

Establishing Heuristic Methods to Improve the Usability of Graphical User Interface Machine Learning Tools

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Usability Evaluation, Machine Learning Tools
• Keywords: GUI ML tools, User Testing, Usability Heuristics, Nielsen Heuristics, Weka, Knime, User Experience, Cognitive Walkthrough, SUS Score

Research Background and Problem

• Problems and Challenges
  • Graphical User Interface Machine Learning (GUI ML) tools are driven by the needs of novice users who often lack a background in machine learning.
  • Current GUI ML tools are complex, presenting unique challenges for novice users in terms of interaction and understanding.
  • There are no dedicated usability heuristic standards to guide and evaluate the design of GUI ML tools.
• Significance
  • As machine learning becomes increasingly accessible to general users, effectively designing and improving the usability of such tools is crucial for advancing the field and promoting its adoption.
• Research Motivation
  • Although the general Nielsen heuristics are widely used for GUI evaluation, they are often too generic to address the specific needs of the complex domain of machine learning tools.
  • The authors aim to improve and extend Nielsen's heuristics to optimize GUI ML tools, providing designers and researchers with a new theoretical and practical framework for usability evaluation.

Solution

• Methods and Approach
  • Propose 14 heuristic methods tailored for GUI ML tools, based on evaluations and user surveys of existing tools (Weka and Knime).
  • The study includes the following steps:
    1. Evaluate the usability of Weka and Knime using existing heuristic methods.
    2. Investigate user needs and challenges with GUI ML tools.
    3. Create a prototype tool aligned with the newly proposed heuristic standards.
    4. Validate the effectiveness of the new heuristics and refine them based on user behavior and feedback.
• Innovations
  • Introduced four new heuristics: Guidance, Credibility, Growth Adaptability, and Context Relevance.
  • Revised and updated five of Nielsen's original heuristics (e.g., language for novice users, commonly used shortcuts).
  • Designed a GUI ML tool prototype specifically for novice users, significantly improving task completion efficiency and error rates through user testing.
• Key Techniques and Implementation Steps
  • Used Cognitive Walkthrough, Heuristic Evaluation, User Testing, and the System Usability Scale (SUS) to evaluate and optimize the tools.
  • Designed an interactive tool prototype using Balsamiq, integrating features such as step-by-step guidance, real-time chat support, and task logging.

Research Outcomes

• Specific Results
  • Heuristic evaluations of Weka and Knime identified critical usability issues (e.g., insufficient feedback, confusing option logic).
  • The newly designed prototype tool successfully reduced user error rates (one-ninth of Weka's error rate) and task completion time (3.9 times faster).
  • The average SUS score improved from Weka's 49 (unacceptable) to 74.62 (acceptable) for the prototype tool.
• Comparison with Existing Solutions
  • The proposed heuristics and prototype clearly outperformed existing Nielsen standards and traditional tools, demonstrating significant improvements in task accuracy and user satisfaction.
  • Enhanced users' understanding and execution of complex tasks such as data cleaning and model deployment.
• Experimental or Evaluation Results
  • Task success rates in all user tests reached 100%, and the average time to complete a task was reduced to 55 seconds.
  • Users emphasized that the new design's "Guidance" and "Credibility" features (e.g., providing interpretable models) were particularly helpful for learning and using the tool.
• Limitations and Future Directions
  • Limitations:
    • The evaluation was limited to two tools (Weka and Knime), which may not fully represent the generalizability to other tools.
    • Some new heuristics (e.g., Growth Adaptability) could not be directly validated in a single experiment and require long-term observation.
  • Future Directions:
    • Develop model guidance and recommendation systems to enhance personalization and trust.
    • Further investigate the fundamental task needs of novice users in interdisciplinary fields.
    • Conduct longitudinal studies to explore growth adaptability and user trust experiences in depth.