MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence

Title of the Paper

MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence

Paper Information

• Topic Area: Intersection of user interface design, programming, and data science, focusing on web automation through Programming by Demonstration (PBD).
• Keywords: Programming by Demonstration (PBD), Web Automation, Data Science, Explainability, UI Design

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Developing web automation scripts requires familiarity with programming languages, HTML structures, and related tools (e.g., Selenium, Scrapy), creating a high learning barrier for non-expert users.
  • Existing Programming by Demonstration (PBD) tools provide limited support for script comprehension, validation, and building user trust. When users identify errors, they often need to start over, leading to high correction costs.
  • Current systems lack effective mechanisms for error diagnosis and correction, making it difficult for users to identify and address potential issues in generated scripts.

• Significance:

  • Web data is crucial for social science researchers and data scientists. Improving the accessibility and user experience of web automation tools can significantly enhance productivity.

• Motivation and Related Work:

  • Programming by Demonstration (PBD) offers hope for enabling non-programming users to achieve web automation without writing code.
  • However, existing tools like Rousillon face limitations such as complex interactions, low user trust, and inconvenient error handling, necessitating improvements.

Solution

• Proposed Method or Solution:

  • Introduce MIWA, a web automation tool supporting mixed-initiative interaction design.
  • Provide a natural language description (NL Description) mechanism for script explanation.
  • Enable users to fine-tune scripts, allowing undo, redo, and editing of demonstrated operations.
  • Incorporate a data anomaly detection feature to help users identify and fix potential issues.

• Innovations:

  • Utilize a syntax-based natural language explanation method to incrementally decompose scripts and translate them into user-friendly descriptions.
  • Implement an incremental script synthesis algorithm with user operation trace editing capabilities to improve script generation efficiency.
  • Offer a visual correspondence feature to help users intuitively understand the relationship between script elements and web page elements.
  • Introduce a detection mechanism for common data anomalies, such as missing data or inconsistent formats, to enhance script robustness.

• Implementation Steps and Key Techniques:

  1. Users perform demonstration operations on the target webpage, which MIWA records.
  2. The system analyzes the operation trace and generates the corresponding web automation script.
  3. Based on user input tables (e.g., CSV files) and recorded operation traces, the script logic is incrementally constructed.
  4. Users can verify whether the script meets expectations through natural language explanations and debugging features, making corrections or adjustments to demonstrations as needed.
  5. The data anomaly detection module monitors script execution in real-time and prompts users to address issues.

Research Outcomes

• Specific Results:

  • Compared to existing tools (e.g., Rousillon), MIWA significantly improves task completion efficiency and is easier for users to understand and trust.
  • User studies show that task completion time with MIWA is halved, and user confidence in the generated scripts is significantly increased.
  • MIWA provides a general web automation framework supporting effective error correction, rapid validation, and cross-page operations.

• Advantages:

  • Enhances script explainability, reducing the comprehension burden for interdisciplinary users.
  • Compared to graphical programming tools, MIWA's natural language explanations are more user-friendly for non-programmers.
  • Offers convenient error correction and debugging mechanisms, improving operational efficiency and reliability.

• Experimental or Evaluation Results:

  • A user study (24 participants) showed that MIWA's average task completion time was 3 minutes and 58 seconds, 55% shorter than Rousillon.
  • Users rated their confidence in MIWA-generated scripts at 6.50 out of 7, significantly higher than the 5.58 confidence score for Rousillon.
  • The system successfully solved 74% of 29 real-world tasks, with an average completion time of under 3 minutes.

• Limitations and Future Directions:

  • Currently, only the top-1 generated script is returned, potentially overlooking correct but lower-ranked scripts. Future work could explore returning multiple candidate scripts for user selection.
  • Does not support dynamic content loading (e.g., infinite scroll lists) or complex interactions on certain dynamic web pages.
  • Data anomaly detection relies on heuristic rules, which may fail to detect some extreme cases.
  • User feedback suggests adding support for operations such as file downloads, image scraping, and time-based tasks.