Exploring Mobile Touch Interaction with Large Language Models

Research Background and Problem

• Identified Issues or Challenges: Currently, when users utilize text editing features powered by large language models (LLMs) on mobile devices, they often need to exit the text writing environment and switch to a conversational AI interface (e.g., ChatGPT app). This approach is cumbersome for space-constrained mobile devices like smartphones, as users must switch between applications to complete text editing tasks, thereby reducing productivity.

• Why This Problem is Important: As LLMs are increasingly used for text generation and editing, the current interaction methods need optimization to better suit portable, small-screen devices while avoiding the inefficiencies of context switching. Additionally, the potential of directly manipulating generative AI remains underexplored, especially in mobile interaction scenarios.

• Research Motivation and Related Work:

  1. Existing research and technologies (e.g., Microsoft's Copilot) primarily integrate text editors with conversational interfaces, but these solutions are less effective on devices with limited screen space, such as smartphones.
  2. Recent explorations (e.g., DirectGPT) have proposed interaction schemes for directly manipulating LLMs using mouse and keyboard, but these approaches are designed for desktop devices and do not align with the nature of touch interactions on mobile devices.
  3. There is a lack of research specifically focused on designing continuous and intuitive gesture interaction models for controlling LLMs in mobile touch environments.

The authors, therefore, propose exploring direct interaction designs for LLMs on mobile devices using touch gestures.

Solution

• Proposed Solution:

  1. Develop a design space specifically tailored for LLM interactions on mobile devices, encompassing dimensions such as input methods, interaction types, text processing approaches, and output presentation.
  2. Design and prototype two touch gesture-to-LLM control mappings:
     • “Spread-to-Generate” Gesture: Generates text through a two-finger spreading motion.
     • “Pinch-to-Shorten” Gesture: Deletes text through a two-finger pinching motion.
  3. Introduce the concept of “word bubbles” through visual feedback design to enhance the interaction experience, providing real-time reflections of the generated text's length and progress.

• Innovative Aspects of the Solution:

  1. Continuous Control Loop: Breaks away from the traditional turn-based conversational interaction model, achieving real-time mapping between user gestures and system-generated text.
  2. Visual Feedback Management: The “word bubbles” design mitigates the irregular latency of LLMs while clearly displaying the length of generated text, making the generation process more intuitive.
  3. New Touch Mappings for Language Models: Assigns more semantic functions to traditional gestures, such as using the spreading gesture to control the length of generated text rather than conventional geometric transformations (e.g., zooming in/out images).

• Implementation Steps and Key Techniques:

  1. Design Space Framework: Organized touch interaction design dimensions through team brainstorming and affinity diagramming.
  2. Prototyping and Iterative Development:
     • The initial prototype supported only simple text generation. Based on user feedback, the team later introduced the “word bubbles” feedback mechanism, more refined sentence selection, and text addition/deletion features.
  3. Evaluating the Design: Refined touch gesture mappings and visual representations through user studies, including experiments to validate the feasibility and user experience improvements of the proposed solution.

Research Outcomes

• Specific Outcomes:

  1. Design Outcomes:
     • Proposed a comprehensive design space framework to systematically explore touch gesture and LLM interaction designs.
     • Implemented a functional prototype supporting two novel gestures (spread-to-generate and pinch-to-shorten).
  2. Experimental Validation:
     • Experiment 1 compared three feedback designs (no feedback, line indicators, and word bubbles), confirming the effectiveness of word bubbles.
     • Experiment 2 demonstrated that gesture-based direct interaction significantly outperformed traditional conversational interfaces (e.g., ChatGPT) in terms of efficiency and user satisfaction.

• Comparative Advantages Over Existing Solutions:

  1. Gesture-based interactions are more efficient than conversational interfaces (average task time reduced by 58%).
  2. Users reported a stronger sense of control when using gestures (significantly higher subjective control ratings).
  3. The “word bubbles” design, with its clear visual feedback and real-time dynamic display, made the system more intuitive, significantly reducing task error rates and cognitive load.

• Experimental or Evaluation Results:

  • In usability metrics (Usability SUS), “word bubbles” scored 85.54 (excellent).
  • In task time, the use of “word bubbles” was significantly faster than conditions with no feedback or line indicators.
  • Compared to ChatGPT, the gesture-based system performed better in task completion speed and subjective task load (NASA-TLX scores).

• Limitations and Future Directions:

  1. Limitations:
     • The study had a small sample size (14 participants), and the experiments were conducted in controlled environments, which may not fully represent real-world scenarios.
     • The prototype system tested only two gestures and did not integrate more complex functionalities or other application scenarios.
  2. Future Directions:
     • Expand the design space to explore new gestures (e.g., rotation for tuning, swiping for reorganization) and other interaction modalities (e.g., device shaking or tilting).
     • Conduct experiments with a broader user base and more complex task requirements.
     • Integrate the design prototype into existing mobile platform applications to test its effectiveness and user acceptance in real-world contexts.

In summary, the authors' research provides significant innovation in the field of mobile touch and LLM interaction, offering valuable guidance and inspiration for future AI interaction design.