DataSpeck: An AI-Driven Human-in-the-Loop System for Automating Transformations in Data Conversion Workflows

Paper Title

DataSpeck: An AI-Driven Human-in-the-Loop System for Automating Transformations in Data Conversion Workflows

Publication Info

• Topic area: Automating data transformation and integration workflows using AI and human-in-the-loop systems.
• Keywords: Data transformation, schema matching, human-in-the-loop, LLMs, data integration, automation, semantic relationships, ETL pipelines, data conversion, AI-driven systems.

Background and Problem

• Problem / challenge: Integrating disparate data sources into pre-existing systems often requires manual data conversion to match the required structure. Existing schema matching and transformation tools leave semantic relationships and complex transformations largely to users, making the process time-consuming and cognitively demanding.
• Significance: Automating data transformation can significantly reduce manual effort, cognitive load, and time required for data integration, which is critical in domains like healthcare, e-commerce, and analytics.
• Motivation and related work: Prior systems like programming-by-example, programming-by-demonstration, and natural language interfaces simplify transformation tasks but still require users to understand data structures and devise transformations. LLMs have shown promise in data-centric tasks but lack end-to-end support for goal-driven workflows. This paper addresses the gap by automating the entire data conversion pipeline while involving users only when necessary.

Solution

• Proposed approach: DataSpeck, an AI-driven human-in-the-loop system, automates the conversion of disparate data sources to fit pre-existing specifications by leveraging LLMs to infer semantic relationships and generate transformation pipelines.
• Novelty:
  1. End-to-end automation of data transformation workflows using AI-driven semantic analysis and pipeline generation.
  2. Human-in-the-loop mechanism for resolving ambiguities through targeted clarification requests.
  3. Confidence-based intervention system to involve users only when necessary.
• Procedure and key techniques:
  • Analyze dataset descriptors to extract structural and semantic information.
  • Infer semantic relationships between source and target datasets using LLMs.
  • Generate and execute transformation pipelines as stepwise Python functions.
  • Employ confidence evaluation to classify outputs as confident, assuming, or insufficient, prompting user input when needed.

Results

• Concrete findings:
  • Automated ~86% of transformations in isolated tasks and ~90% in real-world tasks.
  • Achieved ~93% accuracy in confidence classifications for isolated tasks and ~86% for real-world tasks.
  • User study (N=12) showed participants completed tasks ~53% faster with DataSpeck compared to Microsoft Excel with Copilot.
  • Participants reported significantly lower cognitive load and frustration (NASA-TLX) with DataSpeck.
• Advantage over baselines:
  • DataSpeck outperformed Excel with Copilot by automating strategy inference and requiring fewer user inputs.
  • Reduced task completion time by 44% for basic tasks and 63% for advanced tasks.
• Experiments / evaluation:
  • Technical evaluation: 43 isolated tasks and 5 real-world scenarios tested system automation and confidence mechanisms.
  • User study: Compared DataSpeck with Excel + Copilot across two data conversion tasks, measuring task efficiency, workload, and usability.
• Limitations and future work:
  • Struggles with operations like filtering and splitting due to lack of observable data features.
  • Silent failures in some cases due to LLM context loss or hallucination.
  • Future directions include integrating domain-specific knowledge, collaborative workflows, and enhanced validation mechanisms.

Summary

DataSpeck is an AI-driven human-in-the-loop system that automates data transformation workflows by leveraging LLMs to infer semantic relationships and generate executable pipelines. It automates ~86–90% of transformations while involving users selectively for ambiguities, significantly reducing task completion time and cognitive load compared to traditional tools like Excel with Copilot. The system’s confidence-based intervention mechanism enhances reliability and user trust. Future work aims to address limitations in handling complex transformations, improve validation, and expand domain-specific applications.