Marco: Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

Title of the Paper

Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models

Paper Information

• Subject Areas: Human-Computer Interaction, Information Retrieval, Artificial Intelligence
• Keywords: Document collections, Sensemaking, Large Language Models, Human-AI collaborative systems, Business document workflows

Research Background and Problem Statement

• Identified Problems:

  • Knowledge workers need to extract key information from complex document collections to complete intricate tasks (e.g., reviewing resumes, analyzing contract risks). However, information foraging often involves repetitive and time-consuming manual labor.
  • Current tools generally lack effective support for document collections, focusing more on operations for individual documents.
  • There is a lack of tools to assist workers in handling multiple document collections simultaneously, resulting in significant time spent on extracting and organizing information during the sensemaking process.

• Significance:

  • Efficiently extracting and processing information from document collections is critical for accelerating decision-making; however, current methods may overlook key information, thereby affecting work quality.

• Research Motivation:

  • Addressing the document-related needs in business processes by innovatively leveraging AI to simplify information foraging workflows at the collection level.
  • Exploring the potential of hybrid guided interfaces (combining human and AI interaction) and the use of large language models (LLMs) to assist in this work.

Solution

• Proposed Method/Solution:

  • The system, named Marco, provides an integrated interactive environment, including a Notebook View, Table View, and Document View.
  • The system leverages large language models (LLMs) for information extraction and analysis:
    • Searching for specific keywords or topics.
    • Document-level querying and cross-document integration for question answering.
    • Offering automated document summarization functionality.

• Innovative Features:

  • Focus on collection-centric interaction, reducing the cognitive load of users by avoiding the need to process multiple documents one by one.
  • Supporting users in dynamically expressing information needs through natural language, enabling efficient automation of information foraging, extraction, and comparison.
  • Providing multidimensional interaction references, including integrating results in table format and highlighting sources within documents to support verification.

• Key Technologies and Steps:

  1. Document Preprocessing: After uploading PDFs, the content is extracted, structured into JSON, and semantically vectorized using SentenceTransformers embeddings.
  2. Action Unit Execution: Utilizing zero-shot and few-shot prompting with large language models (e.g., GPT-3.5) to support question answering, searching, and summarization functions.
  3. Multi-Document Problem Decomposition: Employing multi-stage prompting strategies to achieve cross-document semantic synthesis.
  4. AI Recommendations: Analyzing user interaction behavior and suggesting potential follow-up queries to help users explore efficiently.

Research Outcomes

• Specific Results:

  • Improved User Efficiency: In comparative experiments, users completed tasks 16% faster with Marco compared to manual methods, while significantly reducing workload.
  • Enhanced User Experience: Users generally provided high usability scores for the system (average SUS score of 74.5, indicating good usability).
  • Alignment with Professional Scenarios: Design explorations revealed that professional knowledge workers found Marco suitable for real-world needs and appreciated its additional organization and analysis capabilities.

• Comparison with Existing Solutions:

  • Enhanced the efficiency of analyzing and comparing large-scale document collections, whereas traditional tools (e.g., PDF readers) remain limited to individual document operations.
  • Supported verification of information results, increasing user trust and control over LLM-generated answers.

• Experimental or Evaluation Results:

  1. Two groups of user experiments were conducted to test efficiency and applicability:
     • In tasks such as resume screening and service contract comparison, users demonstrated significant advantages in time, effort, and interaction burden when using Marco.
     • Expert feedback indicated that introducing verifiable AI assistance reduced common errors and supplemented workflows.
  2. The system explicitly displayed the sources of model-generated content, strongly supporting verification steps in high-risk tasks.

• Limitations and Future Directions:

  • Limitations:
    • The system currently focuses primarily on text content operations, and task designs are relatively short, requiring further research into compatibility with long-term complex workflows.
    • Potential hallucination issues with LLMs could lead to over-reliance in high-risk decision-making scenarios.
  • Future Directions:
    • Expanding support for multimodal document content (e.g., images, tables).
    • Exploring more complex interaction designs and uncertainty handling tools (e.g., result visualization or cluster analysis) in long-term user scenarios.

Conclusion

Marco demonstrates how AI technologies at the collection level can empower business document workflows by reducing the cost of repetitive labor and tedious organization, thereby enabling users to focus on deeper sensemaking and decision analysis. This research lays the foundation for future design and evaluation of hybrid systems aimed at document processing.