Live in the Loop: Rapid Run-time Feedback for Prompts

Paper Title

Live in the Loop: Rapid Run-time Feedback for Prompts

Publication Info

• Topic area: AI-assisted programming with live feedback mechanisms.
• Keywords: LLMs, live programming, run-time feedback, code generation, exploratory programming, microversioning, prompt refinement, user study, AI assistants, IDE extensions.

Background and Problem

• Problem / challenge: Current AI coding assistants like GitHub Copilot lack mechanisms to provide immediate run-time feedback for generated code, leading to inefficiencies in refining prompts and validating code suggestions.
• Significance: Addressing the "gulf of envisioning" and "gulf of evaluation" in AI-assisted programming can improve the accuracy and intentionality of code generation, reducing errors and iterations.
• Motivation and related work: Prior work on live programming environments and tools like LEAP has demonstrated the benefits of immediate feedback for evaluating code changes. However, these systems focus on code-level interactions rather than run-time results, leaving gaps in prompt refinement and exploration of multiple solutions.

Solution

• Proposed approach: ReFiQ (“Result-first Queries”), a live programming tool that prioritizes run-time results over code and displays multiple results for a single prompt.
• Novelty:
  1. Presentation of run-time values before code suggestions.
  2. Support for exploring multiple interpretations of prompts simultaneously.
  3. Integration of live programming mechanisms like probes to anchor interactions at the run-time data level.
  4. Encouragement of intentional prompt refinement through immediate feedback.
• Procedure and key techniques:
  • Users interact with probes to view live run-time values.
  • Prompts are issued directly referencing run-time data.
  • Multiple code suggestions are generated and executed automatically, displaying their run-time results first.
  • Users select desired behavior based on run-time effects and progressively disclosed code changes.
  • The system supports microversioning to track and revert changes.

Results

• Concrete findings:
  • ReFiQ reduced manual code iterations by 49% compared to GitHub Copilot.
  • Participants issued 9% more prompts and spent 56% more time refining prompts in ReFiQ.
  • Programs created with ReFiQ had significantly fewer errors (p = 0.022) and higher output quality.
• Advantage over baselines:
  • Faster feedback loops for prompt refinement.
  • Higher trust in correctness and output of generated code.
  • Better exploration of solution space through multiple run-time results.
• Experiments / evaluation:
  • Exploratory user study with 8 participants using a within-subjects design.
  • Tasks: OCR preprocessing and topography segmentation.
  • Metrics: task completion, output quality, interaction logs, surveys, and thematic analysis.
• Limitations and future work:
  • Limited validity outside visual domains like image processing.
  • Small sample size (n=8) affects statistical power.
  • Lack of support for undo/redo of prompts and backtracking across iterations.
  • Future work could explore integrating prompts as first-class programming artifacts and enhancing explanations for code suggestions.

Summary

ReFiQ introduces a result-first approach to AI-assisted programming, enabling users to refine prompts and validate code suggestions through immediate run-time feedback. The tool demonstrated reduced manual iterations, higher output quality, and improved exploration of solution space compared to GitHub Copilot. While the study highlighted trade-offs in code comprehension and generalization, ReFiQ's design advances live programming workflows by prioritizing run-time results and supporting intentional decision-making. Future work could address limitations in backtracking and expand applicability to non-visual domains.