Interaction Substrates: Combining Power and Simplicity in Interactive SystemsToday’s graphical user interfaces tend to be either simple but limited, or powerful but overly complex. In order to combine power and simplicity, we introduce Substrates, which act as “places for interaction” where users can manipulate objects of interest in a principled and predictable way. Substrates structure and contain data, enforce user-defined constraints among objects and manage dependencies with other substrates. Users can “tune” and “tweak” these relationships, “curry” specialized tools or abstract relationships into interactive templates. We first define substrates and provide in-depth descriptions with examples of their key characteristics. After explaining how Substrates extend the concept of Instrumental Interaction, we apply a Generative Theory of Interaction approach to analyze and critique existing interfaces and then show how using the concepts of Instruments and Substrates inspired novel design ideas in three graduate-level HCI courses. We conclude with a discussion and directions for future work.2025WMWendy E Mackay et al.Inria, ExSituPrototyping & User TestingCHI
Lorgnette: Creating Malleable Code ProjectionsProjections of computer languages are tools that help users interact with representations that better fit their needs than plain text. We collected 62 projections from the literature and from a design workshop and found that 60% of them can be implemented using a table, a graph or a form. However, projections are often hardcoded for specific languages and situations, and in most cases only the developers of a code editor can create or adapt projections, leaving no room for appropriation by their users. We introduce Lorgnette, a new framework for letting programmers augment their code editor with projections. We demonstrate five examples that use Lorgnette to create projections that can be reused in new contexts. We discuss how this approach could help democratise projections and conclude with future work.2023CGCamille Gobert et al.Knowledge Worker Tools & WorkflowsComputational Methods in HCIUIST
Interaction Knowledge: Understanding the ‘Mechanics’ of Digital ToolsUser interfaces typically feature tools to act on objects and rely on the ability of users to discover or learn how to interact with them. Previous work in HCI has used the Theory of Affordances to explain how users understand the possibilities for action in digital environments. A complementary theory from cognitive neuroscience, Technical Reasoning, posits that users accumulate abstract knowledge of object properties and technical principles known as mechanical knowledge, essential in tool use. Drawing from this theory, we introduce interaction knowledge as the ``mechanical'' knowledge of digital environments. We provide evidence of its relevance by reporting on an experiment where participants performed tasks in a digital environment with ambiguous possibilities for interaction. We analyze how interaction knowledge was transferred across two digital domains, text editing and graphical editing, and conclude that interaction knowledge models an essential type of knowledge for interacting in the digital world.2023MRMiguel A. Renom et al.Université Paris-Saclay, CNRS, InriaVisualization Perception & CognitionUser Research Methods (Interviews, Surveys, Observation)Computational Methods in HCICHI
Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on TouchscreensSuggesting multiple target candidates based on touch input is a possible option for high-accuracy target selection on small touchscreen devices. But it can become overwhelming if suggestions are triggered too often. To address this, we propose SATS, a Suggestion-based Accurate Target Selection method, where target selection is formulated as a sequential decision problem. The objective is to maximize the utility: the negative time cost for the entire target selection procedure. The SATS decision process is dictated by a policy generated using reinforcement learning. It automatically decides when to provide suggestions and when to directly select the target. Our user studies show that SATS reduced error rate and selection time over Shift~\cite{vogel2007shift}, a magnification-based method, and MUCS, a suggestion-based alternative that optimizes the utility for the current selection. SATS also significantly reduced error rate over BayesianCommand~\cite{zhu2020using}, which directly selects targets based on posteriors, with only a minor increase in selection time.2022ZLZhi Li et al.Stony Brook UniversityHand Gesture RecognitionHuman-LLM CollaborationCHI
i-LaTeX: Manipulating Transitional Representations between LaTeX Code and Generated DocumentsDocument description languages such as LaTeX are used extensively to author scientific and technical documents, but editing them is cumbersome: code-based editors only provide generic features, while WYSIWYG interfaces only support a subset of the language. Our interviews with 11 LaTeX users highlighted their difficulties dealing with textually-encoded abstractions and with the mappings between source code and document output. To address some of these issues, we introduce Transitional Representations for document description languages, which enable the visualisation and manipulation of fragments of code in relation to their generated output. We present i-LaTeX, a LaTeX editor equipped with Transitional Representations of formulae, tables, images, and grid layouts. A 16-participant experiment shows that Transitional Representations let them complete common editing tasks significantly faster, with fewer compilations, and with a lower workload. We discuss how Transitional Representations affect editing strategies and conclude with directions for future work.2022CGCamille Gobert et al.Université Paris-SaclayPrototyping & User TestingComputational Methods in HCICHI
Passages: Interacting with Text Across DocumentsA key aspect of knowledge work is the analysis and manipulation of sets of related documents. We conducted interviews with 12 patent examiners and 12 scientists and found that all use specialized tools for managing text from multiple documents across various interconnected activities, including searching, collecting, annotating, organizing, writing and reviewing, while manually tracking their provenance. We introduce Passages, interactive objects that reify text selections that can then be manipulated, reused, and shared across multiple tools. Passages directly supports the above-listed activities as well as fluid transitions among them, e.g. through drag-and-drop across windows. Two user studies show that participants found Passages both elegant and powerful, facilitating their work practices and enabling greater reuse and novel strategies for analyzing and composing documents. We argue that Passages offers a general approach applicable to a wide variety of text-based interactions.2022HHHan L. Han et al.Université Paris-Saclay, CNRS, InriaKnowledge Management & Team AwarenessKnowledge Worker Tools & WorkflowsCHI
Exploring Technical Reasoning in Digital Tool UseThe Technical Reasoning hypothesis in cognitive neuroscience posits that humans engage in physical tool use by reasoning about mechanical interactions among objects. By modeling the use of objects as tools based on their abstract properties, this theory explains how tools can be re-purposed beyond their assigned function. This paper assesses the relevance of Technical Reasoning to digital tool use. We conducted an experiment with 16 participants that forced them to re-purpose commands to complete a text layout task. We analyzed self-reported scores of creative personality and experience with text editing, and found a significant association between re-purposing performance and creativity, but not with experience. Our results suggest that while most participants engaged in Technical Reasoning to re-purpose digital tools, some experienced "functional fixedness." This work contributes Technical Reasoning as a theoretical model for the design of digital tools.2022MRMiguel A. Renom et al.Université Paris-Saclay, CNRS, InriaPrototyping & User TestingComputational Methods in HCICHI
SonicHoop: Using Interactive Sonification to Support Aerial Hoop PracticesAerial hoops are circular, hanging devices for both acrobatic exercise and artistic performance that let us explore the role of interactive sonification in physical activity. We present SonicHoop, an augmented aerial hoop that generates auditory feedback via capacitive touch sensing, thus becoming a digital musical instrument that performers can play with their bodies. We compare three sonification strategies through a structured observation study with two professional aerial hoop performers. Results show that SonicHoop fundamentally changes their perception and choreographic processes: instead of translating music into movement, they search for bodily expressions that compose music. Different sound designs affect their movement differently, and auditory feedback, regardless of type of sound, improves movement quality. We discuss opportunities for using SonicHoop as an aerial hoop training tool, as a digital musical instrument, and as a creative object; as well as using interactive sonification in other acrobatic practices to explore full-body vertical interaction.2021WLWanyu Liu et al.LRI Université Paris-Saclay, STMS IRCAM-CNRS-Sorbonne UniversitéDance & Body Movement ComputingCHI
FileWeaver: Flexible File Management with Automatic Dependency TrackingKnowledge management and sharing involves a variety of specialized but isolated software tools, tied together by the files that these tools use and produce. We interviewed 23 scientists and found that they all had difficulties using the file system to keep track of, re-find and maintain consistency among related but distributed information. We introduce FileWeaver, a system that automatically detects dependencies among files without explicit user action, tracks their history, and lets users interact directly with the graphs representing these dependencies and version history. Changes to a file can trigger recipes, either automatically or under user control, to keep the file consistent with its dependants. Users can merge variants of a file, e.g. different output formats, into a polymorphic file, or morph, and automate the management of these variants. By making dependencies among files explicit and visible, FileWeaver facilitates the automation of workflows by scientists and other users who rely on the file system to manage their data.2020JGJulien Gori et al.Knowledge Management & Team AwarenessKnowledge Worker Tools & WorkflowsUIST
ImageSense: An Intelligent Collaborative Ideation Tool to Support Diverse Human-Computer PartnershipsProfessional designers create mood boards to explore, visualize, and communicate hard-to-express ideas. We present ImageSense, an intelligent, collaborative ideation tool that combines individual and shared work spaces, as well as collaboration with multiple forms of intelligent agents. In the collection phase, ImageSense offers fluid transitions between serendipitous discovery of curated images via ImageCascade, combined text- and image-based Semantic search, and intelligent AI suggestions for finding new images. For later composition and reflection, ImageSense provides semantic labels, generated color palettes, and multiple tag clouds to help communicate the intent of the mood board. A study of nine professional designers revealed nuances in designers' preferences for designer-led, system-led, and mixed-initiative approaches that evolve throughout the design process. We discuss the challenges in creating effective human-computer partnerships for creative activities, and suggest directions for future research.2020JKJanin Koch et al.Teams, Groups, and CreativityCSCW
Textlets: Supporting Constraints and Consistency in Text DocumentsWriting technical documents frequently requires following constraints and consistently using domain-specific terms. We interviewed 12 legal professionals and found that they all use a standard word processor, but must rely on their memory to manage dependencies and maintain consistent vocabulary within their documents. We introduce Textlets, interactive objects that reify text selections into persistent items. We show how Textlets help manage consistency and constraints within the document, including selective search and replace, word count, and alternative wording. Eight participants tested a search-and-replace Textlet as a technology probe. All successfully interacted directly with the Textlet to perform advanced tasks; and most (6/8) spontaneously generated a novel replace-all-then-correct strategy. Participants suggested additional ideas, such as supporting collaborative editing over time by embedding a Textlet into the document to flag forbidden words. We argue that Textlets serve as a generative concept for creating powerful new tools for document editing.2020HHHan L. Han et al.Université Paris-Saclay, CNRS, InriaKnowledge Worker Tools & WorkflowsPrototyping & User TestingCHI
How Relevant is Hick's Law for HCI?Hick's law is a key quantitative law in Psychology that relates reaction time to the logarithm of the number of stimulus-response alternatives in a task. Its application to HCI is controversial: Some believe that the law does not apply to HCI tasks, others regard it as the cornerstone of interface design. The law, however, is often misunderstood. We review the choice-reaction time literature and argue that: (1) Hick's law speaks against, not for, the popular principle that 'less is better'; (2) logarithmic growth of observed temporal data is not necessarily interpretable in terms of Hick's law; (3) the stimulus-response paradigm is rarely relevant to HCI tasks, where choice-reaction time can often be assumed to be constant; and (4) for user interface design, a detailed examination of the effects on choice-reaction time of psychological processes such as visual search and decision making is more fruitful than a mere reference to Hick's law.2020WLWanyu Liu et al.IRCAM Centre Pompidou; Télécom Paris, Institut Polytechnique de Paris; Université Paris-SaclayVisualization Perception & CognitionUser Research Methods (Interviews, Surveys, Observation)CHI
Touchstone2: An Interactive Environment for Exploring Trade-offs in HCI Experiment DesignTouchstone2 offers a direct-manipulation interface for generating and examining trade-offs in experiment designs. Based on interviews with experienced researchers, we developed an interactive environment for manipulating experiment design parameters, revealing patterns in trial tables, and estimating and comparing statistical power. We also developed TSL, a declarative language that precisely represents experiment designs. In two studies, experienced HCI researchers successfully used Touchstone2 to evaluate design trade-offs and calculate how many participants are required for particular effect sizes. We discuss Touchstone2's benefits and limitations, as well as directions for future research.2019AEAlexander Eiselmayer et al.University of ZurichUser Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingComputational Methods in HCICHI
Rethinking Interaction: From Instrumental Interaction to Human-Computer PartnershipsThe extraordinary advances in hardware and networking technology over the past 50 years have not been matched by equivalent advances in software. Today’s interactive systems are fraught with limitations and incompatibilities: they lack interoperability and flexibility for end users. The goal of this workshop is to rethink interaction by identifying frameworks, principles and approaches that break these limitations and create true human-computer partnerships.2018MBMichel Beaudouin-Lafon et al.Univ. Paris-Sud, CNRS, Inria, Université Paris-SaclayHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationCHI
The Perils of Confounding Factors: How Fitts’ Law Experiments can Lead to False ConclusionsThe design of Fitts' historical reciprocal tapping experiment gravely confounds index of difficulty ID with target distance D: Summary statistics for the candidate Fitts model and a competing model may appear identical, and the validity of Fitts' model for some tasks can be legitimately questioned. We show that the contamination of ID by either target distance D or width W is due to the common practices of pooling and averaging data belonging to different distance-width (D,W) pairs for the same ID, and taking a geometric progression for values of D and W. We analyze a case study of the validation of Fitts' law in eye-gaze movements, where an unfortunate experimental design has misled researchers into believing that eye-gaze movements are not ballistic. We then provide simple guidelines to prevent confounds: Practitioners should carefully design the experimental conditions of (D,W), fully distinguish data acquired for different conditions, and put less emphasis on r² scores. We also recommend investigating the use of stochastic sampling for D and W.2018JGJulien Gori et al.Télécom ParisTech, Université Paris-SaclayUser Research Methods (Interviews, Surveys, Observation)Computational Methods in HCICHI
BIGFile: Bayesian Information Gain for Fast File RetrievalWe introduce BIGFile, a new fast file retrieval technique based on the Bayesian Information Gain framework. BIGFile provides interface shortcuts to assist the user in navigating to a desired target (file or folder). BIGFile’s split interface combines a traditional list view with an adaptive area that displays shortcuts to the set of file paths estimated by our computationally efficient algorithm. Users can navigate the list as usual, or select any part of the paths in the adaptive area. A pilot study of 15 users informed the design of BIGFile, revealing the size and structure of their file systems and their file retrieval practices. Our simulations show that BIGFile outperforms Fitchett et al.’s AccessRank, a best-of-breed prediction algorithm. We conducted an experiment to compare BIGFile with ARFile (AccessRank instantiated in a split interface) and with a Finder-like list view as baseline. BIGFile was by far the most efficient technique (up to 44% faster than ARFile and 64% faster than Finder), and participants unanimously preferred the split interfaces to the Finder.2018WLWanyu Liu et al.Telecom ParisTech, Université Paris-Saclay, Univ. Paris-Sud, CNRS, Inria, Université Paris-SaclayInteractive Data VisualizationTime-Series & Network Graph VisualizationVisualization Perception & CognitionCHI
Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-UsabilityVideo prototypes help capture and communicate interaction with paper prototypes in the early stages of design. However, designers sometimes find it tedious to create stop-motion videos for continuous interactions and to re-shoot clips as the design evolves. We introduce Montage, a proof-of-concept implementation of a computer-assisted process for video prototyping. Montage lets designers progressively augment video prototypes with digital sketches, facilitating the creation, reuse and exploration of dynamic interactions. Montage uses chroma keying to decouple the prototyped interface from its context of use, letting designers reuse or change them independently. We describe how Montage enhances video prototyping by combining video with digital animated sketches, encourages the exploration of different contexts of use, and supports prototyping of different interaction styles.2018GLGermán Leiva et al.Video Production & EditingPrototyping & User TestingUIST