DiLogics: Creating Web Automation Programs with Diverse Logics

Title of the Paper

DiLogics: Creating Web Automation Programs With Diverse Logics

Paper Information

• Research Area: Programming Automation, Human-Computer Interaction, Natural Language Processing
• Keywords: Web Automation, PBD (Programming-By-Demonstration), Neuro-Symbolic Programming, Natural Language Processing, GUI Automation, User Experience, Programming Logic, Task Semantics, Automation Error Handling, Human-Machine Collaboration

Research Background and Problems

• Challenges:

  • Web automation tools are typically limited to executing fixed sequences of UI operations, making it difficult to handle tasks with diverse input conditions.
  • Existing tools require domain expertise, making it hard for beginners to quickly create the necessary automation programs.
  • When task inputs involve diverse semantics, current tools struggle to generalize and extend program logic based on such diversity.
  • Manual operations are repetitive, prone to errors, and time-consuming, especially under complex conditional logic.

• Significance:

  • Automation can improve productivity and reduce human errors, but existing tools are insufficiently flexible to meet diverse task requirements.
  • By lowering programming barriers and enhancing tool intelligence, web automation can become more widespread and cater to more practical scenarios.

• Motivation and Related Work:

  • Existing PBD tools (e.g., SemanticOn and Rousillon) are constrained by unified logic, while complex tasks demand smarter and more flexible tools.
  • Neuro-symbolic programming has been proposed as a solution combining the flexibility of neural networks with the precision of symbolic reasoning, but its application in web automation remains underdeveloped.
  • Commercial tools (e.g., UiPath and Taxy AI) offer high-level natural language task specifications but limit users’ ability to modify and refine program logic.

Solution

• Method and Innovations:

  • DiLogics is proposed as a demonstration-based tool that integrates natural language processing (NLP) and neuro-symbolic programming to help non-expert users create diverse web automation programs.
  • NLP is used to semantically segment input data, breaking tasks into manageable steps.
  • By recording user demonstration actions, the system generates GUI operation mappings associated with task semantics.
  • The system infers UI operation logic for similar semantic tasks, enabling generalization across tasks and pages.
  • An error-handling mechanism is integrated, allowing users to edit and optimize logic at any stage while generating programs.

• Implementation Steps and Techniques:

  1. Users upload task input data (e.g., JSON files), and DiLogics performs semantic segmentation to generate task steps.
  2. Using a browser plugin, users demonstrate how to complete tasks via UI operations. The system dynamically parses web content and matches corresponding semantic UI elements.
  3. After two rounds of demonstration, the system generates an automation program and enters semi-automatic mode, allowing users to validate suggested actions.
  4. The system generalizes new task semantics through semantic matching. If entirely new steps are encountered, automation pauses and requires user demonstration.
  5. Users can pause, edit task requirements, and redefine logic at any point in the process, supporting error correction.

• Innovations:

  • Utilizes neuro-symbolic programming to combine symbolic reasoning (rule recognition) and statistical learning (semantic matching), dynamically binding task logic to UI actions.
  • Introduces the concept of "diverse programming logics," enabling programs to handle diverse execution paths based on input data.
  • Provides semantic-enhanced mappings from tasks to UI operations, overcoming limitations of traditional tools reliant solely on page structure.

Research Results

• Specific Outcomes:

  • Developed the DiLogics system and successfully applied it to various common web automation tasks (e.g., restaurant orders, shopping, drug searches, ticketing operations).
  • Validated the system’s effectiveness in task semantic classification, step demonstration, and logic generalization through user studies involving 10 participants and 40 experimental tasks.

• Advantages:

  • Compared to existing tools, DiLogics enables handling diverse task logic without requiring programming knowledge, excelling in semantically complex, multi-condition tasks.
  • The system features an intuitive interaction process, allowing users to continuously optimize programs through natural task descriptions and action demonstrations.
  • Provides flexible error-handling capabilities, such as task redefinition and step re-demonstration.

• Experimental Results:

  • In user experiments, the average task completion time was 8 minutes, with an average task accuracy of 91.2%.
  • The system was rated as easy to use (average user satisfaction score of 6.1/7) and was deemed effective for completing repetitive tasks previously requiring manual operations.

• Limitations and Future Directions:

  • Limited to text-based web pages, unable to handle task elements involving images or graphical user interfaces.
  • Model generalization for new task steps requires two rounds of initial demonstration, not supporting one-shot "zero demonstration."
  • Input data must be structured (e.g., tables), with limited support for unstructured text tasks.

Future Work Directions

• Expanding Multimodal Capabilities: Introduce multimodal models (e.g., CLIP) to enhance understanding of visual information such as webpage images and icons.
• Enhancing Task State Monitoring: Develop real-time webpage state analysis tools to dynamically check task progress and UI status.
• Introducing Task History Reuse: Automatically recommend logic and scripts from previously completed tasks to improve programming efficiency.

This study provides significant insights into improving semantic understanding and user-friendliness in web automation tools, laying the foundation for future applications of artificial intelligence in task automation.