Designing and Evaluating AI Margin Notes in Document Reader Software

Paper Title

Designing and Evaluating AI Margin Notes in Document Reader Software

Publication Info

• Topic area: Integration of AI capabilities into document reader software for enhanced note-taking.
• Keywords: AI margin notes, document reader software, LLMs, note-taking, reading comprehension, user preferences, integration, selection automation, human-AI collaboration.

Background and Problem

• Problem / challenge: Existing AI-enhanced document readers primarily use separate chat interfaces, which are disconnected from the document text. This separation increases cognitive load and reduces efficiency in note-taking and comprehension tasks.
• Significance: Integrating AI capabilities directly into document comments could streamline note-taking, reduce cognitive effort, and improve user experience, especially in educational contexts.
• Motivation and related work: Prior research highlights the benefits of note-taking for learning and the potential of LLMs to support reading comprehension. However, existing systems lack integration with document text and fail to explore the effects of varying levels of human and AI involvement in note-taking.

Solution

• Proposed approach: AI margin notes—comments enhanced with LLM capabilities that are integrated directly into the document text.
• Novelty:
  1. Introduction of AI margin notes as an integrated alternative to chat-based interfaces.
  2. Empirical evaluation of three design parameters: integration, selection automation, and human-AI involvement.
  3. Insights into user preferences, psychological ownership, and workload associated with different AI margin note techniques.
  4. Design implications for improving AI integration in document readers and beyond.
• Procedure and key techniques:
  • Three experiments were conducted:
    1. Comparing AI margin notes to chat-based interfaces for integration.
    2. Evaluating manual vs. automatic text selection for creating AI margin notes.
    3. Exploring six techniques with varying levels of human and AI involvement, including summaries, fill-in-the-blank exercises, and feedback on user-written text.

Results

• Concrete findings:
  • Participants preferred AI margin notes over chat interfaces due to ease of use, reduced interface switching, and better contextual integration.
  • Manual text selection increased psychological ownership and was preferred despite being slower and more effortful.
  • Techniques with more AI involvement (e.g., summaries, fill-in-the-blank exercises) were faster and generally preferred, though they resulted in lower psychological ownership.
  • No significant differences in reading comprehension were observed across techniques, even with varying cognitive demands.
• Advantage over baselines:
  • AI margin notes were ranked higher than chat interfaces for usability and user satisfaction.
  • Manual selection and techniques with moderate AI involvement struck a balance between user control and efficiency.
• Experiments / evaluation:
  • Experiment 1: Compared AI margin notes to chat interfaces (26 participants).
  • Experiment 2: Compared manual vs. automatic text selection (30 participants).
  • Experiment 3: Evaluated six techniques with varying human and AI involvement (32 participants).
  • Metrics included reading comprehension, task duration, psychological ownership, workload, and user preferences.
• Limitations and future work:
  • Results may not generalize to long-term use or educational settings with different constraints.
  • Experimental design lacked ecological validity (e.g., fixed number of notes, isolated techniques).
  • Future work should explore adaptive AI margin notes, integration in other domains, and long-term impacts on learning and note-taking skills.

Summary

This paper introduces AI margin notes as a novel approach to integrating LLM capabilities into document reader software. Through three experiments, it demonstrates that users prefer integrated, manually created AI margin notes over chat-based interfaces and automatic text selection. Techniques with moderate AI involvement were valued for their efficiency and usability, though they reduced psychological ownership. The findings suggest that document readers should offer multiple AI margin note techniques to accommodate diverse user preferences and goals. This work highlights the potential of integrated AI tools to improve reading and note-taking experiences across various contexts.