How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study

Document Title

How Do Data Analysts Respond to AI Assistants? — A Guided Experimental Study

Document Information

• Subject Area: Human-Computer Interaction (HCI), Data Science, Artificial Intelligence Assistants
• Keywords: Data analysis, statistical analysis, AI assistants, code assistants, human-AI interaction, data science tools, planning assistance, guided experiments, LLM-assisted computation

Research Background and Problem

• Problem or Challenge:

  1. During the data analysis process, analysts need to make various complex decisions, such as data cleaning and statistical modeling. These decisions are highly flexible and may lead to conflicting or divergent conclusions among analysts.
  2. Current tools based on large language models (LLMs, such as GitHub Copilot) primarily focus on code execution assistance but lack support for analysis "planning."
  3. The nonlinear and iterative nature of data analysis makes it difficult for existing AI assistants to accurately capture the analyst's current context and needs, such as proposing potential alternative decisions during the planning stage.
  4. Execution preferences in data analysis may lead analysts to overlook the diversity and depth required for high-quality analysis.

• Importance:

  • Data analysis serves as the foundation for scientific research and decision-making. However, deficiencies in analysis planning may result in unreliable outcomes and exacerbate the reproducibility crisis in scientific research.
  • Providing high-quality data analysis suggestions can not only enhance the robustness of analysis results but also help analysts identify potential biases.

• Research Motivation and Related Work:

  • Although the advancements of LLMs have been applied in code generation and auto-completion functionalities, their role in supporting analysts in planning contexts remains underexplored.
  • Research in areas such as "explainable AI" and "human-AI collaboration" reveals new opportunities for designing analysis assistants, but specific design directions in the context of data analysis are still lacking.

Solution

• Method or Solution:

  1. Designed an AI data analysis assistant framework that integrates "planning assistance + execution assistance."
  2. Reviewed literature and analyzed a crowdsourced analysis study to categorize various types of actionable suggestions for the analysis process (e.g., domain background information, operational constructs, model selection, summarized in Table 1).
  3. Conducted a guided experiment (Wizard-of-Oz Study) in which 13 data analysts interacted with an AI assistant simulated by human experimenters to complete specified tasks, observing their reactions and preferences regarding planning and execution assistance.

• Innovations:

  • Proposed a categorization framework for planning suggestion types spanning the entire data analysis lifecycle.
  • Simulated an LLM-driven "suggestion generation–timely delivery" intelligent interaction process in the experiment, focusing on user acceptance behavior during the planning stage.
  • Explored the limitations of current LLM assistants and provided design recommendations for data analysis assistants that go beyond traditional execution assistance perspectives.

• Implementation Steps and Techniques:

  1. Developed a JupyterLab extension as the user interface for interacting with the assistant.
  2. In the guided experiment, participants received real-time suggestions generated by experiment controllers (Wizard).
  3. Conducted qualitative and quantitative analysis on user preferences regarding suggestion types and timing.

Research Outcomes

• Specific Outcomes:

  1. Planning facilitation: The study identified the usefulness of planning-related suggestions, such as high-level planning and high-dimensional variable selection, which helped analysts consider unnoticed decisions.
  2. The experiment demonstrated that planning and execution assistance have respective strengths and weaknesses in terms of timing and task alignment, with planning assistance significantly improving decision quality and depth.
  3. User acceptance of suggestions was influenced by multiple factors, including "content category," "expert background comparison," and "timing accuracy" (Figures 5 and 6 provide detailed analyses of these dynamics).

• Comparison with Existing Solutions and Advantages:

  • Compared to tools that only offer code completion (e.g., ChatGPT, Copilot), the assistant designed in this study integrates both execution and planning assistance while considering dynamic contextual adaptation.
  • Emphasized the "planning advantage" in data analysis tasks rather than direct task completion, significantly enhancing the breadth and depth of analytical decision-making.

• Experimental Results:

  • On average, each analyst received 11.85 suggestions, with planning-related content accounting for 9.85 suggestions. The acceptance rate for planning suggestions that could be embedded into analysis tasks (e.g., variable operationalization, model selection) was 51.6%.
  • Case studies demonstrated that better timing coordination and contextual control could greatly enhance user experience.

• Limitations and Future Directions:

  • Limitations:
    1. The study primarily recruited analysts with high statistical skills, which may limit the generalizability of results to beginners.
    2. The experiment duration was relatively short (approximately 2 hours), making it difficult to capture changes in long-term interaction habits.
    3. The study did not directly evaluate the specific improvement in analysis quality or accuracy due to AI suggestions.
  • Future Directions:
    1. Develop practical systems for long-term use to study behavioral evolution.
    2. Assess the assistant's overall impact on analysis correctness, diversity, and interpretability.
    3. Expand the study sample to include analysts with broader skill backgrounds.

Through the experiments and design presented in this study, the research team provides important insights for the development of future AI planning assistance tools, demonstrating that such assistants can significantly enhance the robustness and reliability of data analysis workflows.