GenieWizard: Multimodal App Feature Discovery with Large Language Models

Research Background and Problem

• What problems or challenges did the authors identify?
  With the emergence of multimodal interactions (e.g., combinations of voice and touch), user behavior and expectations have become more flexible and complex, making it difficult for developers to predict the actions users might want to perform. Even applications using state-of-the-art multimodal frameworks fail to support up to 41% of user requests, negatively impacting user experience and the naturalness of user exploration.

• Why is this problem important?
  While the flexibility of multimodal interaction enhances user experience, it also introduces risks to the development process. If user requests are not supported by the application, users may feel frustrated and stop using the app. This makes early-stage development and testing of multimodal applications even more critical.

• Research Motivation and Related Work
  Current design and prototyping tools (e.g., Figma) support feature discovery for GUI applications during the design phase, but the same strategies are difficult to apply to multimodal applications due to the more complex implementation requirements of multimodal interactions. Additionally, while existing frameworks can use LLMs to parse user commands and automatically adapt them to functionalities, most frameworks still struggle to identify new features that developers have not implemented.

Solution

• What methods or solutions did the authors propose?
  The authors designed and implemented a tool called "GenieWizard," which can identify potential user-expected features during the early stages of application development and provide developers with suggestions in the form of APIs. Its core functionality is based on interaction simulation and feature inference using large language models (LLMs).

• What are the innovative aspects of this solution?

  • Using LLMs to generate user behaviors and interactions, enabling comprehensive simulation of user needs without requiring real user testing.
  • Parsing user behavior into functional code using a zero-shot semantic parser and inferring unimplemented features.
  • A unique "dry-run" technique that performs type inference on unimplemented features through abstract interpretation, avoiding actual calls to developer code.
  • Transforming the user command generation problem into a clustering problem within the development environment, consolidating similar features and providing developers with concrete suggestions.

• What are the implementation steps and key technologies used?

  1. User Interaction Simulation: Providing the program's code framework, using LLMs to generate application descriptions, user roles and scenarios, and simulated dialogues between users and the application.
  2. Feature Inference:
     • Using a zero-shot semantic parser to parse simulated user dialogues into DSL code and infer potentially missing features.
     • Employing the dry-run technique to identify the first unexecutable feature or attribute (unimplemented part).
  3. Suggestion Generation and Integration:
     • Using LLMs to generate annotations for unimplemented features and clustering them; similar features are grouped and specific development suggestions are generated.
     • Presenting suggestions directly to developers via an IDE plugin and tracking the progress of feature implementation.

Research Outcomes

• What specific results were achieved?

  • GenieWizard’s automated generation of user commands covered approximately 71% of the demands found in real user commands.
  • Developers using GenieWizard were able to implement 42% of unsupported features, compared to only 10% without the tool.

• What are the advantages compared to existing solutions?

  • Significantly reduces the need for complex user research in the early stages of development, helping developers identify unimplemented features and quickly improve multimodal applications.
  • The zero-shot parser does not require predefined examples, making it more flexible and cost-effective than few-shot parsers.
  • Provides actionable API-level suggestions, accommodating the complexity of multimodal interaction combinations.

• What are the experimental or evaluation results?

  • User Command Evaluation: GenieWizard-generated commands covered 74% of unsupported real user commands in a food ordering application and 67% in a hotel booking application.
  • Developer Experience: Developers using GenieWizard significantly improved task completion rates (from 10% to 42%) and experienced a notable reduction in cognitive load (as measured by significantly lower NASA-TLX scores). The system's usability score (SUS) reached 77.5, outperforming the baseline score of 41.46.

• Limitations and Future Directions

  • Although the zero-shot parser performs well, it may be influenced by inherent biases in LLM generation, such as gender or cultural biases in population generation and suggestions.
  • GenieWizard has so far only been tested in the first iteration of the development cycle, and its ability to integrate more advanced user feedback has not yet been explored.
  • Future research could extend this feature discovery strategy to other domains, such as the development of AI voice assistants, to further enhance overall user experience.

Conclusion

GenieWizard effectively addresses the challenge of feature discovery in multimodal interaction development by leveraging LLMs to simulate user behavior and perform semantic parsing, generating actionable suggestions for developers. Experiments demonstrated that developers using this tool significantly improved feature implementation rates while reducing cognitive load and development time. Tools like this enable developers to better understand user needs and substantially reduce the proportion of unsupported commands during the development process, providing critical support for the widespread adoption of multimodal applications.