Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models

Document Title

Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models

Document Information

• Domain: Human-Computer Interaction (HCI) and Natural Language Programming Tool Development
• Keywords: Natural language interface, code synthesis, large generative language models, user strategies, adaptive prompts, user experience evaluation

Research Background and Problem

• Problem Description:
  1. Although large generative language models (LLMs) can understand natural language and generate code or other content, users may still encounter obstacles during actual usage.
  2. Specifically, users need to explore and learn how to interact with the model using the correct "syntax," even though the input is in natural language.
  3. Users lack an accurate mental model of the scope of requests the model can fulfill (e.g., complexity or specificity).
• Research Importance: Exploring existing pain points and user experience limitations of natural language programming tools can promote more efficient human-computer collaboration paradigms and lower the entry barriers to programming.
• Research Motivation and Related Work:
  • The introduction of LLMs has made it possible to adjust model outputs through prompts. Existing studies and tools (e.g., GPT-3 and related demonstrations) have shown that generative models can convert natural language into code such as HTML and SQL.
  • Tools like GitHub CoPilot have attempted to integrate code suggestions based on generative models into development environments.

Solution

• Proposed Method and Tool:

  1. Developed a tool named GenLine.
  2. This tool utilizes a 137-billion-parameter generative language model and a set of task-specific prompts to enable the direct conversion of natural language into code.
  3. Provides an interactive interface allowing users to input natural language requests using double brackets, select task-specific prompts, and browse or modify the generated code.
  • For example: [[make this button blue <button>OK</button>]].

• Innovative Aspects of the Solution:

  1. Combines natural language and code input methods, enhancing input flexibility.
  2. Offers dynamic fallback prompts to help users regenerate or adjust input when initial natural language requests fail.
  3. Supports diverse requests (e.g., overly generalized or overly specific tasks).

• Key Technical Details:

  • The tool consistently uses generative models to convert designed prompts into code outputs, enabling conversions from natural language to HTML, JavaScript, and CSS.
  • Implements a few-shot learning prompt system, such as automatically generating code templates from natural language descriptions.
  • Users can adjust the "temperature" (randomness parameter) to influence the diversity of generated results.

Research Results

• User Study:

  1. Study Design: Recruited 14 participants with varying levels of front-end development experience to complete two programming tasks (building a static search page and an interactive flashcard application).
  2. Data Collection: Collected over 7 hours of user operation recordings and post-session interview feedback.
  3. Qualitative and Quantitative Analysis:
     • Task Completion Rate: Approximately 50% of users successfully completed the more complex interactive task (Task 2).
     • Model Output Usage Rate: Less than 50% of generated code was directly accepted.
     • Participants exhibited discrepancies in their mental models regarding the tool's question scope and syntax, requiring significant trial and error.

• Findings and Contributions:

  1. Challenges of using natural language code generation:
     • Users need to learn the tool's "implicit syntax," despite the appearance of natural language input.
     • Uncertainty exists regarding the scope of requests and model performance (e.g., whether it can generate complete functional blocks).
     • Lack of support for referencing input file context or history.
  2. Debugging or correction strategies adopted by users:
     • Rewording requests.
     • Adjusting temperature (modifying generation randomness).
     • Expanding or simplifying input scope.
  3. Reflections on the tool's future potential:
     • Can be used to reduce tedious boilerplate code generation or serve as an auxiliary tool for learning programming.

• Design Recommendations and Improvement Directions:

  1. Provide input suggestions and semantic clarification within the system.
  2. Enhance user control through visualized internal model explanations.
  3. Develop stronger interaction mechanisms, such as decomposing complex requests into subtasks during prompt design.

• Limitations and Future Directions:

  • Limited by model error rates and the implementation of early prototype tools.
  • Future work could test strategy effectiveness on large-scale user data, particularly regarding automatic fallback strategies.