How Notations Evolve: A Historical Analysis with Implications for Supporting User-Defined AbstractionsTraditional human-computer interaction takes place through formally-specified systems like structured UIs and programming languages. Recent AI systems promise a new set of informal interactions with computers through natural language and other notational forms. These informal interactions can then lead to formal representations, but depend upon pre-existing formalisms known to both humans and AI. What about novel formalisms and notations? How are new abstractions created, evolved, and incrementally formalized over time -- and how might new systems, in turn, be explicitly designed to support these processes? We conduct a comparative historical analysis of notation development to identify some relevant characteristics. These include three social stages of notation development: invention & incubation, dispersion & divergence, and institutionalization & sanctification, as well as three functional stages: descriptive, generative, and evaluative. Within and across these stages, we detail several patterns, such as the role of linking and grounding metaphors, dimensions of meaningful variation, and analogical alignment. Finally, we offer some implications for design.2026JZJingyue Zhang et al.Université de montréalGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationParticipatory DesignCHI
Visual Story-Writing: Writing by Manipulating Visual Representations of StoriesWe define "visual story-writing" as using visual representations of story elements to support writing and revising narrative texts. To demonstrate this approach, we developed a text editor that automatically visualizes a graph of entity interactions, movement between locations, and a timeline of story events. Interacting with these visualizations results in suggested text edits: for example, connecting two characters in the graph creates an interaction between them, moving an entity updates their described location, and rearranging events on the timeline reorganizes the narrative sequence. Through two user studies on narrative text editing and writing, we found that visuals supported participants in planning high-level revisions, tracking story elements, and exploring story variations in ways that encourage creativity. Broadly, our work lays the foundation for writing support, not just through words, but also visuals.2025DMDamien Masson et al.Data StorytellingGame AccessibilityAI-Assisted Creative WritingUIST
Semantic Commit: Helping Users Update Intent Specifications for AI Memory at ScaleAs AI agents increasingly rely on memory systems to align with user intent, updating these memories presents challenges of semantic conflict and ambiguity. Inspired by impact analysis in software engineering, we introduce SemanticCommit, a mixed-initiative interface to help users integrate new intent into intent specifications—natural language documents like AI memory lists, Cursor Rules, and game design documents—while maintaining consistency. SemanticCommit detects potential semantic conflicts using a knowledge graph-based retrieval-augmented generation pipeline, and assists users in resolving them with LLM support. Through a within-subjects study with 12 participants comparing SemanticCommit to a chat-with-document baseline (OpenAI Canvas), we find differences in workflow: half of our participants adopted a workflow of impact analysis when using SemanticCommit, where they would first flag conflicts without AI revisions then resolve conflicts locally, despite having access to a global revision feature. Additionally, users felt SemanticCommit offered a greater sense of control without increasing workload. Our findings indicate that AI agent interfaces should help users validate AI retrieval independently from generation, suggesting that the benefits from improved control can offset the costs of manual review. Our work speaks to the need for AI system designers to think about updating memory as a process that involves human feedback and decision-making.2025PVPriyan Vaithilingam et al.Human-LLM CollaborationAI-Assisted Decision-Making & AutomationAlgorithmic Transparency & AuditabilityUIST
Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming SupportAI programming tools enable powerful code generation, and recent prototypes attempt to reduce user effort with proactive AI agents, but their impact on programming workflows remains unexplored. We introduce and evaluate Codellaborator, a design probe LLM agent that initiates programming assistance based on editor activities and task context. We explored three interface variants to assess trade-offs between increasingly salient AI support: prompt-only, proactive agent, and proactive agent with presence and context (Codellaborator). In a within-subject study (N=18), we find that proactive agents increase efficiency compared to prompt-only paradigm, but also incur workflow disruptions. However, presence indicators and interaction context support alleviated disruptions and improved users' awareness of AI processes. We underscore trade-offs of Codellaborator on user control, ownership, and code understanding, emphasizing the need to adapt proactivity to programming processes. Our research contributes to the design exploration and evaluation of proactive AI systems, presenting design implications on AI-integrated programming workflow.2025KPKevin Pu et al.University of Toronto, Department of Computer ScienceHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationPrototyping & User TestingCHI
ChainBuddy: An AI-assisted Agent System for Generating LLM PipelinesAs large language models (LLMs) advance, their potential applications have grown significantly. However, it remains difficult to evaluate LLM behavior on user-defined tasks and craft effective pipelines to do so. Many users struggle with where to start, often referred to as the "blank page problem." ChainBuddy, an AI workflow generation assistant built into the ChainForge platform, aims to tackle this issue. From a single prompt or chat, ChainBuddy generates a starter evaluative LLM pipeline in ChainForge aligned to the user's requirements. ChainBuddy offers a straightforward and user-friendly way to plan and evaluate LLM behavior and make the process less daunting and more accessible across a wide range of possible tasks and use cases. We report a within-subjects user study comparing ChainBuddy to the baseline interface. We find that when using AI assistance, participants reported a less demanding workload, felt more confident, and produced higher quality pipelines evaluating LLM behavior. However, we also uncover a mismatch between subjective and objective ratings of performance: participants rated their successfulness similarly across conditions, while independent experts rated participant workflows significantly higher with AI assistance. Drawing connections to the Dunning–Kruger effect, we discuss implications for the future design of workflow generation assistants regarding the risk of over-reliance.2025JZJingyue Zhang et al.Université de Montréal, Montréal HCI; Mila - Quebec AI InstituteHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationCHI
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesDue to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation. We present a mixed-initiative approach to “validate the validators”— aligning LLM-generated evaluation functions (be it prompts or code) with human requirements. Our interface, EvalGen, provides automated assistance to users in generating evaluation criteria and implementing assertions. While generating candidate implementations (Python functions, LLM grader prompts), EvalGen asks humans to grade a subset of LLM outputs; this feedback is used to select implementations that better align with user grades. A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative nature of alignment. In particular, we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria. What is more, some criteria appear dependent on the specific LLM outputs observed (rather than independent and definable a priori), raising serious questions for approaches that assume the independence of evaluation from observation of model outputs. We present our interface and implementation details, a comparison of our algorithm with a baseline approach, and implications for the design of future LLM evaluation assistants.2024SSShreya Shankar et al.Human-LLM CollaborationUIST
Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable MotivationWe ideate a future design workflow that involves AI technology. Drawing from activity and communication theory, we attempt to isolate the new value that large AI models can provide design compared to past technologies. We arrive at three affordances - dynamic grounding, constructive negotiation, and sustainable motivation - that summarize latent qualities of natural language-enabled foundation models that, if explicitly designed for, can support the process of design. Through design fiction, we then imagine a future interface as a diegetic prototype, the story of Squirrel Game, that demonstrates each of our three affordances in a realistic usage scenario. Our design process, terminology, and diagrams aim to contribute to future discussions about the relative affordances of AI technology with regard to collaborating with human designers.2024PVPriyan Vaithilingam et al.Human-LLM CollaborationDesign FictionDIS
"It's Sink or Swim": Exploring Patients' Challenges and Tool Needs for Self-Management of Postoperative Acute PainPoorly managed postoperative acute pain can have long-lasting negative impacts and pose a major healthcare issue. There is limited investigation to understand and address the unique needs of patients experiencing acute pain. In this paper, we tackle this gap through an interview study with 14 patients who recently underwent postoperative acute pain to understand their challenges in pain self-management and their need for supportive tools. Our analysis identified various factors associated with the major aspects of acute pain self-management. Together, our findings indicated that tools for supporting these patients need to carefully consider information and support delivery to adapt to rapid changes in pain experiences, offer personalized and dynamic assistance that adapts to individual situations in context, and monitor emotion when promoting motivation. Overall, our work provided valuable knowledge to address the less-investigated but highly-needed problem of designing technology for the self-management of acute pain and similar health conditions.2024SZSouleima Zghab et al.Polytechnique MontrealMental Health Apps & Online Support CommunitiesChronic Disease Self-Management (Diabetes, Hypertension, etc.)CHI