A Human-Computer Collaborative Editing Tool for Conceptual Diagrams

Title of the Paper

A Human-Computer Collaborative Editing Tool for Conceptual Diagrams

Paper Information

• Subject Area: Human-Computer Collaboration, Conceptual Diagram Editing, Natural Interaction
• Keywords: Conceptual Diagrams, Natural Content Editing, Multimodal Interaction, Human-Computer Collaboration, Mobile Devices, Editing Tools, Artificial Intelligence, User Research, User Interface, Interaction Design

Research Background and Problem

• Identified Problems or Challenges:

  • Editing tasks (e.g., conceptual diagram editing) involve numerous tedious GUI operations, particularly on mobile devices, leading to inefficiency and poor user experience.
  • Users need to translate their goals into complex operations during editing tasks, resulting in high manual interaction costs.
  • Most current tools either provide option suggestions only or rely entirely on AI to generate tasks, lacking effective collaboration mechanisms.

• Significance:

  • The widespread use of mobile devices has increased the demand for working anytime, anywhere, but traditional GUI interaction methods struggle to adapt to the constraints of mobile scenarios.
  • Enhancing content editing efficiency and effectiveness is crucial for productivity in office work and various other fields.

• Research Motivation and Related Work:

  • Current human-computer collaboration systems mainly focus on annotation tools or AI generation tools, with little optimization for specific editing tasks.
  • Existing multimodal interaction methods have not adequately addressed the issue of understanding users' ambiguous instructions (e.g., uncertain positions or attributes).
  • A novel conceptual diagram editing tool could optimize task allocation by integrating AI and multimodal interaction, thereby reducing user interaction burdens.

Solution

• Proposed Method or Solution:

  • Designed and implemented a novel human-computer collaborative editing tool, SGDiag, which employs multimodal interaction (voice, gestures) combined with artificial intelligence to support fast and accurate diagram editing.
  • The tool divides tasks so that humans and computers focus on their respective strengths: users describe goals with high-level ambiguous instructions, and the system processes these instructions to compute and recommend detailed actions.
  • Provides candidate solutions and displays the logic behind them, enabling users to select the optimal solution.

• Innovations:

  • Proposed a new task allocation strategy: determining the functional distribution between humans and computers by observing user and human assistant interactions.
  • Introduced AI and natural multimodal interaction into diagram editing tasks, significantly reducing user interaction burdens.
  • The system intelligently completes missing content, resolves instruction conflicts, and recommends optimal solutions from multiple alternatives.

• Implementation Steps and Key Technologies:

  1. Function Design:
     • Defined task allocation through user behavior studies.
     • The system supports completing ambiguous or missing information and resolving potential conflicts in user instructions.
  2. System Architecture:
     • Implemented modules for instruction parsing, solution generation and recommendation, and intent prediction.
  3. Interaction Design:
     • Supports a complete interaction process, including pre-command prediction, mid-command parsing, and result recommendation.
     • Visualizes element relationships and the logic of topological relationships in candidate solutions.
  4. Content Modeling:
     • Mathematically modeled user diagram content and its topological relationships (e.g., linear constraints on element positions).

Research Outcomes

• Specific Results:

  • Identified feasible task allocation between users and the system through research.
  • Successfully developed the collaborative editing tool SGDiag for conceptual diagrams, supporting multimodal interaction and operable on mobile devices.
  • User experiments demonstrated that SGDiag improved editing efficiency (average increase of 32.75%) and result quality (score improvement of 21.89%) compared to existing tools.

• Comparative Advantages Over Existing Solutions:

  • Compared to traditional GUI-based tools, it reduces the need for precise user input, offering natural and effective interaction.
  • Capable of handling ambiguities in user instructions and intelligently recommending resolutions for conflicts.
  • Outperforms existing commercial applications (e.g., PowerPoint for Android) on mobile devices.

• Experimental or Evaluation Results:

  • Two user studies were conducted with 16 and 24 participants, respectively. The results showed:
    1. SGDiag successfully generated reasonable candidate solutions, with users selecting the system's top recommendation in most cases.
    2. It significantly outperformed the baseline tool, PowerPoint, in terms of editing time and user experience.
    3. Users were able to quickly make decisions when accessing suggested intents and selecting candidate solutions, and they rated the tool highly for ease of use and intelligence.

• Limitations and Future Directions:

  • Limitations:
    • Voice interaction in public settings may pose privacy concerns.
    • The current tool lacks modeling for relationships between non-positional attributes and has limited capability for parsing semantic content.
    • Further comparisons with more professional devices equipped with hardware support (e.g., drawing tablets) are needed.
  • Future Directions:
    • Extend to more tasks and devices, adding support for other interaction modes (e.g., gaze tracking, head gestures).
    • Enhance semantic understanding of content and optimize modeling of non-linear relationships.
    • Leverage machine learning to improve the robustness of voice command parsing.

In summary, this paper proposes an innovative diagram editing tool, SGDiag, which significantly enhances efficiency and user experience through human-computer collaboration, natural interaction, and advanced AI technologies.