Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Models

Document Title

Selenite: Scaffold Online Sensemaking with Comprehensive Overviews Elicited from Large Language Models

Document Information

• Subject Area: Human-Computer Collaboration and Information Processing
• Keywords: Human-Computer Collaboration, Sensemaking, Large Language Models, Natural Language Processing, User Interface Design, Information Retrieval, Data Visualization

Research Background and Problem

• Problems and Challenges:
  The authors highlight that when processing information and making decisions in unfamiliar domains, users need to compare different options and evaluate multiple criteria. This process is often time-consuming and labor-intensive, prone to information overload and decision errors due to a lack of structured frameworks. Additionally, existing sensemaking tools face a "cold start" problem, struggling to generate comprehensive, unbiased, and structured overviews.

• Significance:
  Information overload and unguided sensemaking processes directly impact the quality and efficiency of user decisions, which is critically important in contexts such as business, education, and research.

• Motivation and Related Work:
  Prior studies and formative research by the authors reveal that providing a global overview at the start of the sensemaking process significantly helps users understand the domain and optimize decision-making. However, existing technologies often rely heavily on user input or generate biased and incomplete overviews, failing to support user search and comprehension effectively. Against this backdrop, the authors aim to explore the use of large language models to generate high-quality domain overviews and dynamic sensemaking support.

Solution

• Method and Solution:
  The authors propose a system called Selenite, which leverages large language models like GPT-4 to generate comprehensive overviews of global options and criteria, helping users initiate the sensemaking process. Selenite dynamically adapts to user behavior by providing real-time annotations and summaries at the page and paragraph levels, as well as recommending key areas for further exploration to enhance users' efficiency and comprehension.

• Innovations:

  1. Pioneering the use of GPT-4 to generate "cold start" global criteria overviews for unfamiliar domains.
  2. Introducing an interactive design that bridges global and local perspectives, including automatic option identification, paragraph-level annotations, and dynamic content analysis.
  3. Proposing a novel human-computer interaction UX/UI design beyond traditional conversational generation interfaces.

• Implementation Steps and Techniques:

  1. Automatic Topic Identification and Criteria Generation: Using GPT-4 to extract general domain criteria and descriptions.
  2. Real-Time Paragraph Analysis: Combining the BART model for natural language inference to annotate options and criteria mentioned in paragraphs.
  3. Dynamic Search Recommendations: Analyzing user behavior and employing graph-based algorithms to recommend supplementary exploration content.
  4. User Interface Design: Providing a sidebar to display options and criteria, supporting dynamic navigation and annotations.

Research Outcomes

• Specific Results:
  Selenite significantly improved users' information processing speed (reducing time by 36.3%), increased the number of criteria identified by users (by 90.4%), and enhanced comprehension quality (precision of criteria identification improved from 78.4% to 98.8%, recall improved from 30.4% to 73.0%).

• Comparison with Existing Solutions:
  Compared to traditional tools, Selenite can autonomously generate structured overviews without supervision, while conveniently supporting dynamic content analysis and search recommendations, greatly reducing users' cognitive load.

• Experiments or Evaluation Results:
  The effectiveness and user experience of Selenite were validated through three studies: intrinsic evaluation (accuracy and coverage analysis), usability evaluation (efficiency and reading depth comparison), and open-ended case studies (qualitative feedback analysis). All demonstrated its significant advantages.

• Limitations and Future Directions:

  1. Inter-criteria Support: Further integrating hierarchical relationships and correlations among criteria into model generation.
  2. Domain Adaptability: Exploring the integration of domain-specific knowledge bases (beyond language models) with LLMs.
  3. Long-Term Effectiveness Studies: Conducting research on the long-term impact in real user scenarios.
  4. Motivational Design: Designing interaction mechanisms to encourage users to read original texts and avoid over-reliance on summaries.

Conclusion

Selenite offers an advanced LLM-based sensemaking support tool that significantly enhances users' reading efficiency and comprehension quality through global overviews and dynamic interactions. This research validates the potential of this approach and provides a reference for designing future LLM-supported learning and exploration systems.