ImaginationVellum: Generative-AI Ideation Canvas with Spatial Prompts, Generative Strokes, and Ideation HistoryWe introduce ImaginationVellum, a multi-modal spatial canvas for early-stage visual ideation and concept sketching with generative AI. The resulting system supports a unique style of human-AI co-creation where the canvas is the prompt. This means that ImaginationVellum employs the entire 2D canvas as an active prompt space, where spatial arrangement, proximity, and composition of diverse content elements - inking, text, images, and intermediate results - steer generative visual outcomes. As a technical probe, ImaginationVellum contributes a set of spatially-grounded direct manipulation tools for iterative visual ideation. In particular, we introduce Generative Strokes - freeform strokes that spatially modulate generation and prompt-parameters (articulated along multiple latent semantic or stylistic dimensions). These techniques afford rapid traversal of design spaces via convergence, divergence, re-composition, blending, and remixing of concepts. We detail the system architecture, design rationale, proximity-dependent intent tags for localized control, and methods for spatial prompting and varying output along spatial gradients. Temporal replay and visualization of provenance make ideation trajectories actionable, turning the design process itself into an artifact that supports reflection-in-action and revisitation of design decisions. We report insights from a preliminary study of how users construct, steer, and revisit ideas using spatial prompts, and discuss tradeoffs in modulating spatially-dependent content generation.2025NMNicolai Marquardt et al.Generative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsUIST
AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose ToolsChat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody "prompts" as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents (Reflection-in-intent) as well as the range of AI-model responses (Reflection-in-response) to inform design "moves" towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM’s to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents.2025NRNathalie Henry Riche et al.Microsoft ResearchGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
Style Blink: Exploring Digital Inking of Structured Information via Handcrafted Styling as a First-Class ObjectStructured note-taking forms such as sketchnoting, self-tracking journals, and bullet journaling go beyond immediate capture of information scraps. Instead, hand-drawn pride-in-craftmanship increases perceived value for sharing and display. But hand-crafting lists, tables, and calendars is tedious and repetitive. To support these practices digitally, Style Blink (“Style-Blocks+Ink”) explores handcrafted styling as a first-class object. Style-blocks encapsulate digital ink, enabling people to craft, modify, and reuse embellishments and decorations for larger structures, and apply custom layouts. For example, we provide interaction instruments that style ink for personal expression, inking palettes that afford creative experimentation, fillable pens that can be “loaded” with commands and actions to replace menu selections, techniques to customize inked structures post-creation by modifying the underlying handcrafted style-blocks and to re-layout the overall structure to match users' preferred template. In effect, any ink stroke, notation, or sketch can be encapsulated as a style-object and re-purposed as a tool. Feedback from 13 users show the potential of style adaptation and re-use in individual sketching practices.2022HRHugo Romat et al.MicrosoftGraphic Design & Typography ToolsCreative Coding & Computational ArtCHI
AirConstellations: In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware ArmaturesAirConstellations supports a unique semi-fixed style of cross-device interactions via multiple self-spatially-aware armatures to which users can easily attach (or detach) tablets and other devices. In particular, AirConstellations affords highly flexible and dynamic device formations where the users can bring multiple devices together in-air – with 2-5 armatures poseable in 7DoF within the same workspace – to suit the demands of their current task, social situation, app scenario, or mobility needs. This affords an interaction metaphor where relative orientation, proximity, attaching (or detaching) devices, and continuous movement into and out of ad-hoc ensembles can drive context-sensitive interactions. Yet all devices remain self-stable in useful configurations even when released in mid-air. We explore flexible physical arrangement, feedforward of transition options, and layering of devices in-air across a variety of multi-device app scenarios. These include video conferencing with flexible arrangement of the person-space of multiple remote participants around a shared task-space, layered and tiled device formations with overview+detail and shared-to-personal transitions, and flexible composition of UI panels and tool palettes across devices for productivity applications. A preliminary interview study highlights user reactions to AirConstellations, such as for minimally disruptive device formations, easier physical transitions, and balancing "seeing and being seen" in remote work.2021NMNicolai Marquardt et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Knowledge Management & Team AwarenessUbiquitous ComputingUIST
Sketchnote Components, Design Space Dimensions, and Strategies for Effective Visual Note TakingSketchnoting is a form of visual note taking where people listen to, synthesize, and visualize ideas from a talk or other event using a combination of pictures, diagrams, and text. Little is known about the design space of this kind of visual note taking. With an eye towards informing the implementation of digital equivalents of sketchnoting, inking, and note taking, we introduce a classification of sketchnote styles and techniques, with a qualitative analysis of 103 sketchnotes, and situated in context with six semi-structured follow up interviews. Our findings distill core sketchnote components (content, layout, structuring elements, and visual styling) and dimensions of the sketchnote design space, classifying levels of conciseness, illustration, structure, personification, cohesion, and craftsmanship. We unpack strategies to address particular note taking challenges, for example dealing with constraints of live drawings, and discuss relevance for future digital inking tools, such as recomposition, styling, and design suggestions.2021RZRebecca Zheng et al.University College London, MumbliInteractive Data VisualizationData StorytellingUser Research Methods (Interviews, Surveys, Observation)CHI
SurfaceFleet: Exploring Distributed Interactions Unbounded from Device, Application, User, and TimeKnowledge work increasingly spans multiple computing surfaces. Yet in status quo user experiences, content as well as tools, behaviors, and workflows are largely bound to the current device—running the current application, for the current user, and at the current moment in time. SurfaceFleet is a system and toolkit that uses resilient distributed programming techniques to explore cross-device interactions that are unbounded in these four dimensions of device, application, user, and time. As a reference implementation, we describe an interface built using Surface Fleet that employs lightweight, semi-transparent UI elements known as Applets. Applets appear always-on-top of the operating system, application windows, and (conceptually) above the device itself. But all connections and synchronized data are virtualized and made resilient through the cloud. For example, a sharing Applet known as a Portfolio allows a user to drag and drop unbound Interaction Promises into a document. Such promises can then be fulfilled with content asynchronously, at a later time (or multiple times), from another device, and by the same or a different user.2020FBFrederik Brudy et al.Distributed Team CollaborationKnowledge Worker Tools & WorkflowsUIST
Tilt-Responsive Techniques for Digital Drawing BoardsDrawing boards offer a self-stable work surface that is continuously adjustable. On digital displays, such as the Microsoft Surface Studio, these properties open up a class of techniques that sense and respond to tilt adjustments. Each display posture—whether angled high, low, or somewhere in-between—affords some activities, but not others. Because what is appropriate also depends on the application and task, we explore a range of app-specific transitions between reading vs. writing (annotation), public vs. personal, shared person-space vs. task-space, and other nuances of input and feedback, contingent on display angle. Continuous responses provide interactive transitions tailored to each use-case. We show how a variety of knowledge work scenarios can use sensed display adjustments to drive context-appropriate transitions, as well as technical software details of how to best realize these concepts. A preliminary remote user study suggests that techniques must balance effort required to adjust tilt, versus the potential benefits of a sensed transition.2020HRHugo Romat et al.Knowledge Worker Tools & WorkflowsNotification & Interruption ManagementUIST
Dear Pictograph: Investigating the Role of Personalization and Immersion for Consuming and Enjoying VisualizationsMuch of the visualization literature focuses on assessment of visual representations with regard to their effectiveness for understanding data. In the present work, we instead focus on making data visualization experiences more enjoyable, to foster deeper engagement with data. We investigate two strategies to make visualization experiences more enjoyable and engaging: personalization, and immersion. We selected pictographs (composed of multiple data glyphs) as this representation affords creative freedom, allowing people to craft symbolic or whimsical shapes of personal significance to represent data. We present the results of a qualitative study with 12 participants crafting pictographs using a large pen-enabled device and while immersed within a VR environment. Our results indicate that personalization and immersion both have positive impact on making visualizations more enjoyable experiences.2020HRHugo Romat et al.Université Paris-Saclay, CNRS, Inria, LRI & Microsoft ResearchImmersion & Presence ResearchData StorytellingVisualization Perception & CognitionCHI
DataSelfie: Empowering People to Design Personalized Visuals to Represent Their DataMany personal informatics systems allow people to collect and manage personal data and reflect more deeply about themselves. However, these tools rarely offer ways to customize how the data is visualized. In this work, we investigate the question of how to enable people to determine the representation of their data. We analyzed the Dear Data project to gain insights into the design elements of personal visualizations. We developed DataSelfie, a novel system that allows individuals to gather personal data and design custom visuals to represent the collected data. We conducted a user study to evaluate the usability of the system as well as its potential for individual and collaborative sensemaking of the data.2019NKNam Wook Kim et al.Harvard UniversityInteractive Data VisualizationData StorytellingCHI
Comparing Effectiveness and Engagement of Data Comics and InfographicsThis paper compares the effectiveness of data comics and infographics for data-driven storytelling. While infographics are widely used, comics are increasingly popular for explaining complex and scientific concepts. However, empirical evidence comparing the effectiveness and engagement of infographics, comics and illustrated texts is still lacking. We report on the results of two complementary studies, one in a controlled setting and one in the wild. Our results suggest participants largely prefer data comics in terms of enjoyment, focus, and overall engagement and that comics improve understanding and recall of information in the stories. Our findings help to understand the respective roles of the investigated formats as well as inform the design of more effective data comics and infographics.2019ZWZezhong Wang et al.University of EdinburghInteractive Data VisualizationData StorytellingCHI
SpaceInk: Making Space for In-Context AnnotationsWhen editing or reviewing a document, people directly overlay ink marks on content. For instance, they underline words, or circle elements in a figure. These overlay marks often accompany in-context annotations in the form of handwritten footnotes and marginalia. People tend to put annotations close to the content that elicited them, but have to compose with the often-limited whitespace. We introduce SpaceInk, a design space of pen+touch techniques that make room for in-context annotations by dynamically reflowing documents. We identify representative techniques in this design space, spanning both new ones and existing ones. We evaluate them in a user study, with results that inform the design of a prototype system. Our system lets users concentrate on capturing fleeting thoughts, streamlining the overall annotation process by enabling the fluid inverleaving of space-making gestures with freeform ink.2019HRHugo Romat et al.Knowledge Worker Tools & WorkflowsPrototyping & User TestingUIST
ActiveInk: (Th)Inking with DataDuring sensemaking, people annotate insights: underlining sentences in a document or circling regions on a map. They jot down their hypotheses: drawing correlation lines on scatterplots or creating personal legends to track patterns. We present ActiveInk, a system enabling people to seamlessly transition between exploring data and externalizing their thoughts using pen and touch. ActiveInk enables the natural use of pen for active reading behaviors, while supporting analytic actions by activating any of these ink strokes. Through a qualitative study with eight participants, we contribute observations of active reading behaviors during data exploration and design principles to support sensemaking.2019HRHugo Romat et al.Microsoft ResearchInteractive Data VisualizationData StorytellingCHI
DataToon: Drawing Dynamic Network Comics With Pen + Touch InteractionComics are an entertaining and familiar medium for presenting compelling stories about data. However, existing visualization authoring tools do not leverage this expressive medium. In this paper, we seek to incorporate elements of comics into the construction of data-driven stories about dynamic networks. We contribute DataToon, a flexible data comic storyboarding tool that blends analysis and presentation with pen and touch interactions. A storyteller can use DataToon rapidly generate visualization panels, annotate them, and position them within a canvas to produce a visually compelling narrative. In a user study, participants quickly learned to use DataToon for producing data comics.2019NKNam Wook Kim et al.Microsoft Research & Harvard UniversityInteractive Data VisualizationData StorytellingCreative Coding & Computational ArtCHI
Design Patterns for Data ComicsData comics for data-driven storytelling are inspired by the visual language of comics and aim to communicate insights in data through visualizations. While comics are widely known, few examples of data comics exist and there has not been any structured analysis nor guidance for their creation. We introduce data-comic design-patterns, each describing a set of panels with a specific narrative purpose, that allow for rapid storyboarding of data comics while showcasing their expressive potential. Our patterns are derived from i) analyzing common patterns in infographics, datavideos, and existing data comics, ii) our experiences creating data comics for different scenarios. Our patterns demonstrate how data comics allow an author to combine the best of both worlds: spatial layout and overview from infographics as well as linearity and narration from videos and presentations.2018BBBenjamin Bach et al.University of EdinburghInteractive Data VisualizationData StorytellingCHI
Animated Edge Textures in Node-Link Diagrams: a Design Space and Initial EvaluationNetwork edge data attributes are usually encoded using color, opacity, stroke thickness and stroke pattern, or some combination thereof. In addition to these static variables, it is also possible to animate dynamic particles flowing along the edges. This opens a larger design space of animated edge textures, featuring additional visual encodings that have potential not only in terms of visual mapping capacity but also playfulness and aesthetics. Such animated edge textures have been used in several commercial and design-oriented visualizations, but to our knowledge almost always in a relatively ad hoc manner. We introduce a design space and Web-based framework for generating animated edge textures, and report on an initial evaluation of particle properties - particle speed, pattern and frequency - in terms of visual perception.2018HRHugo Romat et al.LRI, Univ. Paris-Sud, CNRS, INRIA, Université Paris-Saclay, TecknowMetrixTime-Series & Network Graph VisualizationData PhysicalizationCHI
DataInk: Direct and Creative Data-Oriented DrawingCreating whimsical, personal data visualizations remains a challenge due to a lack of tools that enable for creative visual expression while providing support to bind graphical content to data. Many data analysis and visualization creation tools target the quick generation of visual representations, but lack the functionality necessary for graphics design. Toolkits and charting libraries offer more expressive power, but require expert programming skills to achieve custom designs. In contrast, sketching affords fluid experimentation with visual shapes and layouts in a free-form manner, but requires one to manually draw every single data point. We aim to bridge the gap between these extremes. We propose DataInk, a system supports the creation of expressive data visualizations with rigorous direct manipulation via direct pen and touch input. Leveraging our commonly held skills, coupled with a novel graphical user interface, DataInk enables direct, fluid, and flexible authoring of creative data visualizations.2018HXHaijun Xia et al.University of TorontoInteractive Data VisualizationCHI
Collaborative Dynamic Queries: Supporting Distributed Small Group Decision-makingCommunication is critical in small group decision-making processes during which each member must be able to express preferences to reach consensus. Finding consensus can be difficult when each member in a group has a perspective that potentially conflicts with those of others. To support groups attempting to harmonize diverse preferences, we propose Collaborative Dynamic Queries (C-DQ), a UI component that enables a group to filter queries over decision criteria while being aware of others’ preferences. To understand how C-DQ affects a group’s behavior and perception in the decision-making process, we conducted 2 studies with groups who were prompted to make decisions together on mobile devices in a dispersed and synchronous situation. In Study 1, we found showing group preferences with C-DQ helped groups to communicate more efficiently and effectively. In Study 2, we found filtering candidates based on each member’s own filter range further improved a groups’ communication efficiency and effectiveness.2018SHSungsoo (Ray) Hong et al.University of WashingtonDistributed Team CollaborationKnowledge Management & Team AwarenessCHI