How Humans Communicate Programming Tasks in Natural Language and Implications For End-User Programming with LLMs

Research Background and Problem

• Identified Issues or Challenges: This study explores how to use natural language to describe Information Processing Tasks (IPTs) and facilitate communication between humans and large language models (LLMs). It identifies challenges in understanding and describing programming tasks in terms of language and logical expression, particularly for users without programming experience. While recent AI tools can translate natural language into code, effective communication barriers remain.
• Significance: Interacting with AI through natural language to express programming tasks holds the potential to lower the learning barrier of traditional programming languages for non-technical users. Research in this area can provide a foundation for designing more efficient and user-friendly interfaces.
• Research Motivation and Related Work: The motivation stems from the potential of LLMs to solve real-world programming problems and the need to design systems that improve user-LLM interaction. Related work has primarily focused on AI-assisted programming tools (e.g., GitHub Copilot) and graphical user interfaces, but the broader scope of natural language interaction with programming tasks remains underexplored.

Solution

• Proposed Method or Solution: The authors conducted two user studies to analyze human-to-human and human-to-LLM communication of IPTs, focusing on the impact of factors such as programming experience, use of examples, and interactivity on conveying complex tasks.
  • Study 1: Compared communication effectiveness between human users and between humans and LLMs.
  • Study 2: Focused on optimizing task descriptions when users directly interact with LLMs.
• Innovations:
  • Systematic analysis of the characteristics of natural language descriptions for programming tasks.
  • Comparison of task communication performance between programmers and non-programmers.
  • Investigation of the impact of interactivity and task examples on communication quality.
  • Analysis of LLM performance in directly answering questions versus generating code.
• Implementation Steps and Key Techniques:
  • User studies involved roles of senders (task describers) and receivers (task interpreters).
  • Experimental design examined the effects of providing examples and allowing user interaction on communication outcomes.
  • Tested whether LLMs could directly answer questions or generate code based on natural language descriptions.
  • Linear regression analysis was used to evaluate the impact of variables (e.g., programming experience, interaction conditions, task complexity) on communication effectiveness.

Research Findings

• Specific Findings:
  • Programming experience improved senders' understanding of tasks but had minimal impact on receivers; non-programmers were also capable of effectively describing and understanding IPTs.
  • Providing examples significantly improved task description quality, but the proportion of users utilizing examples was low, indicating that non-technical users may struggle to create useful examples.
  • Interactivity had limited contribution to improving task communication, especially when senders' descriptions were unclear.
  • GPT-4 performed better in directly answering test cases than in generating code, particularly for tasks involving complex logic and edge cases.
• Advantages Compared to Existing Solutions:
  • Provided detailed experimental data on human-AI communication in natural language tasks, offering a more comprehensive perspective than prior research focused on code generation applications.
  • Investigated specific communication elements (e.g., the impact of different interaction or example formats), offering guidance for designing AI programming tools for non-technical users.
• Experimental or Evaluation Results:
  • Significant correlation was found between senders' task understanding and receivers' performance.
  • GPT-4 and other LLMs excelled in directly answering questions but frequently misunderstood user intent.
  • Human senders often omitted definitions or conveyed incorrect information processing logic, which was a primary cause of LLM failures.
• Limitations and Future Directions:
  • Limitations: The representativeness of the selected IPT tasks remains uncertain; real-world use cases may differ from the experimental design.
  • Future Directions: Enhance users' ability to generate high-quality task descriptions, such as through "task description check" tools; further optimize LLM strategies for inferring user intent; explore whether LLMs can provide functional feedback instead of redundant information.

In summary, this paper systematically reveals the complexity of describing programming tasks in natural language through experimental studies and proposes important design principles, such as the necessity of examples and the appropriate level of interactivity. These findings have significant implications for developing programming tools tailored to non-technical users.