Steering Semantic Data Processing With DocWrangler

DOI

Unstructured text has long been difficult to analyze automatically at scale. Large language models (LLMs) now make it possible to perform “semantic data processing,” where common data operators like map, reduce, and filter are powered by LLMs instead of traditional code. However, building these pipelines is challenging: users need to understand their data in order to build effective pipelines, but at the same time, they need pipelines to extract the information that would help them understand their data. They also have to navigate the quirks and inconsistencies of LLMs. We introduce DocWrangler, a mixed-initiative development environment designed for semantic data processing. DocWrangler offers three new features to help bridge gaps between users, their data, and their pipelines: (i) In-Situ User Notes, which lets users inspect, annotate, and keep track of observations across documents and LLM outputs; (ii) LLM-Assisted Prompt Refinement, which helps turn user notes into improved data processing operations; and (iii) LLM-Assisted Operation Decomposition, which detects when tasks are too complex for the LLM and suggests how to break them down. Our evaluation combines a user study with 10 participants and a public-facing deployment (at docetl.org/playground) with over 1,500 recorded sessions. The results show that users develop systematic strategies for working with semantic data pipelines—such as turning open-ended prompts into classifiers for easier validation, and deliberately using vague prompts to explore their data or the LLM’s behavior.