Automatic Macro Mining from Interaction Traces at Scale

Title of the Paper

Automatic Macro Mining from Interaction Traces at Scale

Paper Information

• Domain: HCI (Human-Computer Interaction), Macro Mining, Mobile UI Automation
• Keywords: User Tasks, Macros, Large Language Models, Mobile User Interface, Task Automation, Trace Mining, LLM, Downstream Applications

Research Background and Problem

• Problem and Challenges:

  • "Macros" in mobile applications, as an essential component of task automation, typically require users to perform multiple steps. However, due to the involvement of multiple views and contextual dependencies, these macros are challenging to extract at scale from interaction logs.
  • Existing methods (e.g., programming by example) struggle to capture high-level complex macros.
  • Even with manual or crowdsourced task generation, covering all functionalities of mobile applications remains challenging.

• Significance: Automatically extracting macros is critical for understanding user experience, task automation, and knowledge sharing. Abstracting and automating these macros can more efficiently support interaction design and development.

• Motivation:

  • With the advancement of large language models (LLMs), exploring their potential in analyzing mobile interaction traces can provide new avenues for macro extraction.

Solution

• Method/System:

  • Task Discovery: Extracting potential user tasks from the interaction UI trees of mobile applications using LLMs.
  • Action Mapping: Inferring specific UI actions required to complete tasks using LLMs.
  • Parameter Extraction: Extracting additional information (e.g., titles or timestamps) needed to complete tasks.
  • Action Merging and Path Optimization: Constructing an interaction graph based on multiple interaction traces and optimizing redundant user steps using shortest path algorithms.
  • Macro Replayer (Macro Executor): Automatically replaying extracted macros to validate and demonstrate automation functionality.

• Technical Core:

  • Leveraging the "chain-of-thought" reasoning capability of LLMs to construct semantically meaningful high-level tasks step by step.
  • Employing dynamic trace merging and graph traversal to optimize user action paths, abstracting optimal execution plans from diverse traces.

• System Data Sources:

  1. Querying real user and random trace datasets, including the RICO and Rehearsal datasets.
  2. Each trace is recorded with UI hierarchy structures, screenshots, and user action information.

• System Features:

  • Does not rely on manual input or specialized application code.
  • Generated macros include natural language descriptions and are both replayable and extendable to multiple tasks.

Research Outcomes

• Summary of Results:

  • Automatically extracted 23,777 semantic and efficient macros from mobile application traces, derived from 4,189 traces (an average of 8.49 macros per app).
  • Macro extraction on the Rehearsal dataset demonstrated higher depth coverage.
  • The proposed system significantly reduced unnecessary steps, e.g., reducing macro path lengths by 43.6% on the RICO dataset.
  • 76.7% of extracted macros were successfully executed in real-world Android emulator environments.

• Innovations:

  • First to demonstrate that LLMs can "semantically interpret" tasks from UI traces while generating directly usable automated workflows.
  • Extracted macros surpass the task coverage of existing datasets like MoTIF, enhancing interaction analysis capabilities.
  • Experiments showed that model-generated descriptions achieved a Rouge-L score of 0.47 compared to human tasks, indicating strong correlation.

• Limitations:

  • Merging nodes may lead to path errors due to duplicate UI IDs.
  • Assumes operations are stateless, ignoring some UI behaviors dependent on dynamic contexts.

• Future Work:

  • Improve the uniqueness and accuracy of node identification; introduce context-based and learning-based frameworks.
  • Consider state-dependent operation execution logic.
  • Expand to more types of UI tasks and domains to enhance generalization capabilities.

Outlook

• Application Prospects:
  • Interaction Task Understanding and Modeling: The dataset can support training intelligent agents for UI navigation and completing complex tasks.
  • Interaction Automation: Combining natural language task selectors and replayers powered by language models can enhance task execution automation.
  • Knowledge Sharing: Useful for generating tutorials and automated guidance, especially for uncovering functional paths that users may not have discovered.