Enabling Conversational Interaction with Mobile UI using Large Language Models

Title of the Paper

Enabling Conversational Interaction with Mobile UI using Large Language Models

Paper Information

• Domain: Human-Computer Interaction (HCI), Mobile User Interfaces (UI), Large Language Models (LLMs)
• Keywords: Large Language Models, Conversational Interaction, Mobile UI, Few-shot Learning, Textual Representation of UI, Automated Screen Summarization

Research Background and Problem Statement

• Problems or Challenges:

  • Current conversational agents require extensive task-specific datasets and training models to generate interactions for various mobile user interface (UI) tasks, resulting in high development costs and time consumption.
  • Existing intelligent assistants (e.g., Google Assistant) have limited capabilities in recognizing screen content and answering mobile UI-based questions for complex tasks.
  • Mobile UI tasks are diverse, and most existing studies can only handle single interaction scenarios, lacking broad adaptability.

• Significance:

  • Conversational interaction has the potential to enhance user experiences with mobile devices through natural language, especially for users with mobility or visual impairments.
  • Achieving unified multi-task conversational interaction capabilities in a lightweight manner can significantly lower the barrier to developing language-interaction agents.

• Motivation and Related Work:

  • Pre-trained large language models (e.g., GPT-3, PaLM) have demonstrated the potential of few-shot learning, enabling adaptation to various language tasks without specialized training.
  • The challenge lies in adapting large language models for graphical user interfaces (GUIs) and enabling conversational tasks based on mobile screens.

Solution

• Methods or Solutions:

  • Utilize large language models (LLMs) with a set of few-shot learning-based prompting techniques to adapt the models for mobile UI tasks.
  • Propose an algorithm to convert hierarchical data from mobile UIs into HTML format, providing text inputs understandable by the model.
  • Define four main task scenarios based on user and intelligent agent interaction contexts:
    1. Screen Question-Generation
    2. Screen Summarization
    3. Screen Question-Answering
    4. Mapping Instruction to UI Action

• Innovations:

  • Textual Representation of Screen Data: Propose using HTML syntax to represent hierarchical data from mobile screens, including class attributes, content descriptions, and resource identifiers.
  • Chain-of-Thought Prompting: Guide the model to generate intermediate logical reasoning processes to complete complex tasks.
  • Few-shot Learning Paradigm: Achieve significant adjustments to the language model using only a few task examples, avoiding complex model retraining.

• Implementation Steps and Key Techniques:

  1. Convert hierarchical data from mobile UIs into HTML format (e.g., <p>, <button> tags) to simulate web UI representation.
  2. Design a complete structure of few-shot prompts based on task characteristics (e.g., input screen, logical reasoning, output answers).
  3. Build and run experiments using the PaLM large language model to evaluate its performance across the four scenarios.

Research Findings

• Specific Results:

  • Developed a set of language task prompting techniques for mobile UIs and designed a sample generation algorithm.
  • In the four task scenarios, the experimental PaLM model achieved performance comparable to specialized training models even with minimal data examples.
  • First exploration in the literature of Screen Question-Generation and Screen Question-Answering tasks.

• Comparison with Existing Solutions:

  • For certain tasks (e.g., Screen Summarization), human evaluation results outperformed existing models (Screen2Words).
  • In the "Mapping Instruction to UI Action" task, while not surpassing specialized training models like Seq2Act, the method still demonstrated competitiveness.

• Experimental Results:

  • Screen Question-Generation Task: Syntax correctness score nearly perfect (4.98/5.0), significantly better than rule-based template methods.
  • Screen Summarization Task: Slightly inferior based on automated metrics, but superior in human annotations (LLM-generated summaries deemed more accurate for 64.1% of screens).
  • Screen Question-Answering: 2-shot LLM achieved an overall answer relevance rate of 84.5%, significantly outperforming the baseline model DistilBERT.
  • Mapping Instruction to UI Action: 1-shot LLM achieved 74.69% partial match accuracy.

• Limitations and Future Directions:

  • Limitations:
    • Currently relies solely on hierarchical representation of mobile UIs, without utilizing pixel or other modality information.
    • Limited support for complex multi-task or multi-turn conversations.
    • Generated content may exhibit uncertainty (i.e., "hallucination phenomenon").
  • Future Directions:
    • Integrate visual information to improve screen representation and extend to more complex interaction scenarios (e.g., multi-screen collaboration).
    • Enhance model inference efficiency (e.g., using model distillation and compression techniques) to support real-time applications.
    • Further explore improving the customization and controllability of large language models.

Conclusion

This study demonstrates the potential of leveraging pre-trained large language models to rapidly enable conversational interaction on mobile UIs through few-shot learning. The proposed prompting techniques allow quick adaptation to multi-task interactions with simple examples, lowering development barriers and driving innovation in language interaction applications.