Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Way

Research Background and Problem

• Identified Issues or Challenges: Data analysts frequently need to iterate between data transformation and chart design during exploratory data analysis. However, existing AI-driven visualization tools often require analysts to describe complex target visualizations in detail through text in a single attempt. This not only demands high user expertise but also lacks support for branching and backtracking, leading to inefficiencies in accomplishing complex tasks.
• Significance: Exploratory tasks in data analysis typically require nonlinear, iterative exploration to uncover potential patterns and insights. Current tools often fall short in supporting such iterative workflows, imposing limitations on users.
• Research Motivation and Related Work: Current AI tools (e.g., systems that generate code based on natural language) are not well-suited for iterative environments. Meanwhile, multi-turn chat-based interaction tools may produce erroneous results due to insufficient management of historical context. This study aims to address these challenges by enhancing iterative chart creation through multimodal interaction and data management.

Solution

• Proposed Solution: The authors developed Data Formulator 2 (Df2), which combines a graphical user interface (GUI) with natural language input to support iterative chart creation. The main features of Df2 include:
  1. Multimodal Chart Builder: Allows users to specify fields through drag-and-drop and typing while using concise natural language instructions to describe target designs.
  2. Data Thread View: Helps users navigate their iterative history, branch into new designs, and reuse previous chart designs.
  3. AI-Powered Data Transformation Generator: Automatically converts user intent into data transformation code, such as filtering, aggregation, ranking, etc.
• Innovations:
  1. By combining GUI and natural language, Df2 reduces the difficulty of precisely expressing complex tasks through detailed text while improving interaction accuracy and flexibility.
  2. The data thread manages the iterative process centered on data versions, enabling users to branch, backtrack, and reuse content more easily, optimizing workflows in exploratory analysis.
• Implementation Steps:
  1. Users initialize chart designs through the chart builder by specifying existing or desired fields.
  2. The system automatically generates Vega-Lite visualization specifications and uses AI to produce the required data transformation code.
  3. Users navigate and manage analysis directions through the data thread, easily making adjustments or branching.
  4. The system provides various mechanisms to help users validate results, including data tables, code explanations, and visual presentations.

Research Outcomes

• Specific Achievements:
  1. Developed the Df2 tool, featuring a multimodal chart builder, multi-branch data threads, and AI-driven efficient data transformation.
  2. In a user study, Df2 successfully helped users complete two sets of complex data exploration tasks, involving 16 iterative visualization tasks (most requiring complex data transformations).
  3. Users developed different iterative and validation workflows based on their preferences, effectively accomplishing data exploration tasks.
• Advantages:
  1. Compared to traditional methods (e.g., handwritten code or GPT-based conversational tools), Df2 lowers the skill threshold and input cost for users.
  2. Data threads overcome the limitations of linear conversations, supporting nonlinear branching and context management.
  3. Multimodal interaction enables users to achieve complex tasks quickly and accurately without requiring detailed natural language descriptions.
• Experiment and Evaluation Results:
  1. In the study, 8 participants accurately reproduced 16 chart tasks from exploratory sessions, demonstrating the effectiveness of Df2's design.
  2. With minimal prompts, participants developed their own iterative strategies, including adopting "broad" or "deep" branching tree structures and selectively backtracking or making incremental adjustments to correct errors.
  3. During the experiment, users primarily relied on generated charts, data tables, and code explanations for validation, reflecting their recognition of the tool's transparency and intuitiveness.
• Limitations and Future Directions:
  1. Limitations:
     • The study tasks focused on replicating professional analysts' exploratory workflows, without examining user behavior in open-ended analysis scenarios.
     • The long-term learning and adaptation effects of users with the tool were not fully explored.
  2. Future Directions:
     • Extend Df2 into a recommendation support tool to assist users in discovering potential exploration directions.
     • Enhance Df2's interaction capabilities for chart editing tasks (e.g., color scheme adjustments, sequence sorting).
     • Introduce proactive clarification mechanisms, enabling the AI to ask precise questions when encountering ambiguous or incomplete user input, reducing the need for subsequent user validation efforts.