Dango: A Mixed-Initiative Data Wrangling System using Large Language Model

Research Background and Issues

• Issues or Challenges:
  Research indicates that data wrangling is a time-consuming, complex, and error-prone stage in the data science workflow, accounting for 80% of data scientists' working time. Existing tools face the following limitations:

  1. Limited to single-table operations, unable to handle multi-table tasks.
  2. Only support user demonstrations to specify transformation intentions, making it difficult to process complex logic or conditional operations.
  3. While natural language descriptions (NL) are concise, they are prone to misunderstandings or generating incorrect operations.
     Additionally, automated tools based on large language models (LLMs) often misinterpret user intentions, lack effective mechanisms to correct or clarify errors, and may generate code that is difficult for users to understand or verify.

• Significance:
  Data wrangling is an indispensable part of data science, and improving the efficiency and accuracy of tools is crucial for enhancing practical work efficiency and reducing user burden. The widespread demand for multi-table data operations further highlights the necessity of tool improvement.

• Research Motivation and Related Work:
  The authors reviewed various interactive tools and programming-by-demonstration (PBD) methods and found that these approaches significantly help data scientists improve efficiency. Furthermore, leveraging the natural language understanding capabilities of LLMs can address the ambiguity of intention expression in existing tools. However, there is currently a lack of hybrid interaction designs to address issues such as intention clarification, complex multi-table operations, and result validation.

Solution

• Method or Solution:
  The authors proposed a hybrid interactive tool—Dango, designed to optimize data wrangling tasks by combining various interaction designs. Its core features include:

  1. Integration of demonstrations and natural language input to support rich intention expression.
  2. Using LLMs to actively generate clarification questions (CQ) to help users clarify their intentions.
  3. Providing step-by-step natural language explanations to assist users in understanding script behavior and supporting direct editing to modify script content.
  4. Offering a data provenance view to track changes in table versions.

• Innovations:

  1. Clarification Question Generation: Actively asking questions to resolve ambiguous intentions rather than solely relying on user descriptions, reducing error rates and intention misunderstandings.
  2. Step-by-Step Natural Language Explanation: Avoiding "hallucination errors" in LLM-generated documentation, making scripts more comprehensible and enabling users to precisely locate issues.
  3. Hybrid Interaction Design: Integrating demonstrations, natural language input, and tool feedback to enable collaboration between users and the system for complex tasks.
  4. Multi-Table Operation Support: Extending DSL language syntax to accommodate multi-table scenarios and advanced data operations such as statistical testing.

• Implementation Steps and Key Technologies:

  • Intention Analysis Phase:
    1. Capturing user demonstration history.
    2. Using LLMs to analyze user input across multiple tables and generate clarification questions (CQ) or summarize clear intentions.
  • DSL Script Generation:
    1. LLMs decompose tasks based on user intentions and generate step-by-step script plans.
    2. Validating the generated DSL syntax to ensure correctness and avoid hallucination errors.
  • Program Verification and Optimization:
    1. Translating DSL scripts into structured natural language explanations to assist user validation.
    2. Allowing users to edit step-by-step explanations or regenerate scripts for optimization.
    3. Providing a data provenance view to visually track modifications.

Research Outcomes

• Specific Outcomes:
  The authors validated Dango's efficiency through experiments and user studies:

  1. It helps users complete a wide range of data tasks (e.g., multi-table merging, conditional filtering, statistical analysis).
  2. It significantly improves user success rates, error correction times, and confidence in script generation.

• Advantages Compared to Existing Solutions:
  Compared to traditional tools and other LLM systems, Dango demonstrates the following advantages:

  1. Users achieved a 100% task success rate (Condition C), whereas baseline tools (Condition A) had failure cases.
  2. Average task completion time reduced by 45% (Condition C compared to Condition A), with clarification questions effectively reducing the frequency of LLM-generated errors.
  3. Particularly for high-complexity tasks, multi-table operations and intention clarification features showed significant efficacy.

• Experimental or Evaluation Results:

  1. User Testing: 38 participants completed 7 data wrangling tasks under three tool design conditions and provided subjective evaluations based on NASA TLX.
     • Under the optimal design condition (C), the average task completion time was 3 minutes 30 seconds (45% shorter than Condition A, 32% shorter than Condition B).
     • Condition C had the lowest rate of hallucination errors in script generation (0.18%), compared to 0.65% for Condition A.
  2. Generalized Complex Task Assessment: Extended testing on 24 tasks, all of which were successfully completed.

• Limitations and Future Directions:

  1. DSL Functionality Limitations: The current DSL language does not support implementing complex control flows (e.g., loops), which may limit its ability to handle higher complexity tasks.
  2. Lack of Real-World Environment: Experimental conditions were relatively controlled and did not simulate collaborative factors and data scale in real-world work environments.
  3. User Preparedness Challenges: Some users still face difficulties in expressing complex intentions, suggesting future optimization of natural language descriptions and interaction design.
  4. Cross-Modal Interaction Expansion: Future work could integrate image annotation or gesture operations to assist in complex data processing scenarios.

Through comprehensive evaluation, Dango demonstrates significant contributions to improving the effectiveness and user experience of data wrangling tools, paving the way for future integrated human-computer interaction designs.