SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation

Research Background and Problem

• What problems or challenges did the authors identify?

  The authors focus on understanding and summarizing user behavior in interactive systems to identify potential user goals. Current goal recognition methods often rely on predefined, fixed goal sets, which fail to accommodate diverse user needs or identify emerging goals. Additionally, the complexity and high variability of interactive behaviors increase the difficulty of accurate prediction and recognition.

• Why is this problem important?

  Understanding user goals is crucial for designing smarter interactive systems, which can support task automation, predict users' next actions, and optimize interface design, thereby improving the efficiency and usability of human-computer interaction.

• Research Motivation and Related Work

  Inspired by text and image summarization tasks in the natural language processing field, the authors propose a framework that summarizes interactive behaviors into natural language sentences. This approach overcomes the limitations of fixed goal classification, supports open-set goal recognition, and provides greater generality and flexibility.

Solution

• What methods or solutions did the authors propose?

  The authors proposed a method called SummAct, a hierarchical interaction behavior summarization approach based on large language models (LLMs). SummAct summarizes users' low-level interaction actions into intermediate-level subgoals, followed by generating high-level overall goal summaries.

• What are the innovations of this solution?

  1. Open-Set Goal Recognition: Unlike traditional methods, SummAct is not limited to predefined goal sets and can handle dynamically evolving and unseen user goals.
  2. Hierarchical Summarization: By generating intermediate-level subgoals, it better conveys semantic information and handles complex, multi-stage interactive behaviors.
  3. UI Element Weighting Mechanism: During fine-tuning, a weighting mechanism ensures that detailed content of interactive interface elements is preserved in the goal summaries, enhancing precision and accuracy.

• What are the implementation steps? What key technologies were used?

  1. Subgoal Generation: Using professionally annotated data, intermediate-level subgoals are generated through in-context learning with a frozen pre-trained LLM.
  2. Overall Goal Summarization: Fine-tuning the LLM generates the final goal summaries, while applying a UI element weighting mechanism. By adjusting loss function weights, interaction details are retained.
  3. Model Training Details: Training was conducted on the lightweight open-source model Mistral-7B, using optimization and learning rate adjustment strategies to ensure efficient model performance.

Research Outcomes

• What specific results were achieved?

  SummAct demonstrated significantly better performance than baseline methods and various ablated versions on two representative interaction datasets (Mind2Web and MoTIF). The generated summaries accurately revealed the hidden goals behind user behaviors.

• What advantages does it have over existing solutions?

  • The hierarchical design of SummAct significantly enhances the model's interpretability and its ability to handle complex interactive behaviors.
  • Its open-set goal recognition capability addresses the limitations of traditional classification methods in real-world scenarios.
  • The UI element weighting mechanism ensures that summaries align more closely with the semantic details of actual user behaviors.

• What were the experimental or evaluation results?

  • SummAct achieved the best performance in embedding semantic similarity (cosine similarity) and multiple N-gram-based text similarity evaluation metrics (BLEU, ROUGE, METEOR), with an embedding similarity score of 0.842.
  • Compared to weaker baseline models (directly using pre-trained LLMs), SummAct's performance improved by up to 240%.

• Limitations and Future Directions

  1. Limitations:
     • Currently, it only processes interaction behaviors at the UI element level and does not utilize pixel-level interaction data, such as touchscreen trajectories.
     • Its ability to handle noisy data has not been fully evaluated, though real-world user behaviors may include non-goal-related actions.
  2. Future Directions:
     • Explore integrating vision-language models to process GUI screenshots and capture more visual details.
     • Address more complex multi-level goal summarization, involving potential cognitive demand transformation issues.
     • Investigate more robust model designs to handle deviations in user behavior caused by operational errors.

Through this innovative interaction behavior summarization method, the authors provide an important new tool for developing smarter and more intuitive interactive systems, while showcasing the immense potential of incorporating advanced language models in the field of HCI.