Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks

Title of the Paper

Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks

Paper Information

• Domain: Human-Computer Interaction, Educational Technology, Augmented Reality
• Keywords: Augmented Reality, Explorable Explanations, Interactive Paper, Augmented Textbooks, Authoring Interface

Research Background and Problem

• Problem or Challenge:
  • Current textbooks primarily offer static explanations, leaving students to passively absorb information without opportunities for interaction or exploration.
  • Creating explorable explanations often requires extensive programming expertise, making it difficult for non-technical users (e.g., teachers and students) to design interactive content tailored to their needs.
• Significance:
  • Explorable explanations help students gain deeper understanding of abstract concepts through interaction and exploration, particularly in subjects like mathematics and physics that benefit from visualization.
  • Transforming static textbooks into interactive media can revolutionize traditional educational methods.
• Research Motivation and Related Work:
  • Existing tools (e.g., GeoGebra, MathPad2) have lowered the barrier for creating educational media but still require technical skills.
  • Leveraging augmented reality (AR) and machine learning technologies to enhance the interactivity of traditional textbooks opens new possibilities for education.

Solution

• Main Approach:
  • Propose an AR-based tool, "Augmented Math," powered by machine learning, enabling users to transform static math textbooks into interactive explorable explanations without requiring programming expertise.
• Innovations:
  • Eliminates the need for programming from scratch by automatically extracting text, formulas, and graphics using OCR and computer vision technologies and converting them into interactive content.
  • Users can interact with extracted content (e.g., formulas and graphics) via mobile AR or desktop interfaces to create dynamic and personalized textbooks.
• Implementation Steps:
  1. Scanning: Capture math textbooks and extract formulas and graphics using OCR and computer vision technologies.
  2. Selection: Users select extracted content (e.g., formulas, variables, graphics).
  3. Binding: Users bind formulas to graphics.
  4. Manipulation: Drag or modify variable values to observe dynamic responses.
  5. Updating: Real-time calculations update graphics and formulas.
  • Key Technologies Used: OCR (Google Cloud, MathPix, CnSTD), computer vision (OpenCV-based), and WebAR development frameworks (A-Frame).

Research Outcomes

• Specific Results:
  • Developed a prototype system for creating AR-enhanced textbooks, supporting five enhancement features: dynamic values, interactive graphics, relationship highlighting, concrete examples, and step-by-step hints.
  • Successfully evaluated the system's effectiveness through two user studies:
    1. Preliminary User Testing (N=11): Confirmed that the AR interface was more engaging, while the desktop interface performed better in usability.
    2. Expert Interviews (N=5): Highlighted the low cost of content creation and adaptability, making it suitable for self-learning and classroom teaching.
• Advantages and Comparison:
  • Compared to static textbooks and videos, the system allows users to directly interact with formulas and graphics, enhancing intuitive understanding.
  • Provides non-technical users with an easy way to create explorable content without requiring complex programming skills.
• Experimental Results:
  • Interactive graphics and relationship highlighting features were highly appreciated during user testing.
  • The desktop version of the system achieved a System Usability Score (SUS) of 81.82 (SD=13.56), outperforming the mobile AR version, which scored 75 (SD=15.32).
• Limitations and Future Directions:
  • The current system still encounters errors in extracting complex formulas and visuals (e.g., interference from decorative or auxiliary lines), requiring improvements in formula and graphic extraction accuracy.
  • The AR user interface faces challenges such as hand tremors and small font sizes, which could be addressed through zoom functionality or projection mapping techniques.
  • Potential expansion into other educational content areas (e.g., language, physics, music) and exploration of diverse AR interfaces (e.g., mixed reality head-mounted devices).