Friction in Processual Ethics: Reconfiguring Ethical Relations in Interdisciplinary ResearchFriction -- disagreement and breakdown -- is an omnipresent aspect of conducting interdisciplinary research yet is rarely presented in formal research reporting. We analyse a performance-led research process where professional dancers with different disabilities explored how to improvise with an industrial robot, with the support of an interdisciplinary team of human-computer and human-robot interaction researchers. We focus on one site of friction in our research process; how to dance -- safely -- with robots? By presenting our research process, we exemplify the different ways in which we encountered this friction and how we reconfigured the research process around it. We contribute five ways in which we arrived at a generative ethical outcome, which may be helpful in productively engaging with friction in interdisciplinary collaboration.2025RGRachael Garrett et al.KTH Royal Institute of Technology, Media Technology and Interaction DesignHuman-Robot Collaboration (HRC)Technology Ethics & Critical HCIDance & Body Movement ComputingCHI
Sensitive Pictures: Emotional Interpretation in the MuseumMuseums are interested in designing emotional visitor experiences to complement traditional interpretations. HCI is interested in the relationship between Affective Computing and Affective Interaction. We describe Sensitive Pictures, an in the museum, experience an emotional story while viewing them, and self-report their response. A subsequent interview with a portrayal of the artist employs computer vision to estimate emotional responses from facial expressions. Visitors are given a souvenir postcard visualizing their emotional data. A study of 132 members of the public (39 interviewed) illuminates key themes: designing emotional provocations; capturing emotional responses; engaging visitors with their data; a tendency for them to align their views with the system’s interpretation; and integrating these elements into emotional trajectories. We consider how Affective Computing can hold up a mirror to our emotions during Affective Interaction.2022SBSteve Benford et al.University of NottinghamInteractive Data VisualizationVisualization Perception & CognitionMuseum & Cultural Heritage DigitizationCHI
Articulating Soma Experiences using TrajectoriesIn this paper, we reflect on the applicability of the concept of trajectories to soma design. Soma design is a first-person design method which considers users' subjective somatic or bodily experiences of a design. Due to bodily changes over time, soma experiences are inherently temporal. Current instruments for articulating soma experiences lack the power to express the effects of experiences on the body over time. To address this, we turn to trajectories, a well-known concept in the HCI community, as a way of mapping this aspect of soma experience. By showing trajectories through a range of dimensions, we can articulate individual experiences and differences in those experiences. Through analysis of a set of soma experience designs and a set of temporal dimensions within the experiences, this paper demonstrates how trajectories can provide a practical conceptual framing for articulating the temporal complexity of soma designs.2021PTPaul Tennent et al.University of NottinghamFull-Body Interaction & Embodied InputCHI
Producing Liveness: The Trials of Moving Folk Clubs Online During the Global PandemicThe global pandemic has driven musicians online. We report an ethnographic account of how two traditional folk clubs with little previous interest in digital platforms transitioned to online experiences. They followed very different approaches: one adapted their existing singaround format to video conferencing while the other evolved a weekly community-produced, pre-recorded show that could be watched together. However, despite their successes, participants ultimately remained unable to ‘sing in chorus’ due to network constraints. We draw on theories of liveness from performance studies to explain our findings, arguing that HCI might orientate itself to online liveness as being co-produced through rich participatory structures that dissolve traditional distinctions between live and recorded and performer and audience. We discuss how participants appropriated existing platforms to achieve this, but these in turn shaped their practices in unforeseen ways. We draw out implications for the design and deployment of future live performance platforms.2021SBSteve Benford et al.University of NottinghamDigital Art Installations & Interactive PerformanceInteractive Narrative & Immersive StorytellingCHI
Translations and Boundaries in the Gap Between HCI Theory and Design PracticeThe gap between research and design practice has long been a concern for the HCI community. In this paper, we explore how different translations of HCI knowledge might bridge this gap. A literature review characterizes the gap as having two key dimensions?one between general theory and particular artefacts and a second between academic HCI research and professional UX design practice. We report on a five-year engagement between HCI researchers and a major media company to explore how a particular piece of HCI research, the trajectories conceptual framework, might be translated into UX practice. We present various translations of this framework and fit them into the gap we previously identified. This leads us to refine the idea of translations, suggesting that they may be led by researchers or co-produced with practitioners as boundary objects. We consider the benefits of each approach.2020RVRaphael Velt et al.User Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingDIS
VRtefacts: Performative Substitutional Reality with Museum ObjectsWe explore how a combination of manipulations and transitions can extend Substitutional Reality to create a highly personal Virtual Reality experience. Our design aimed to meet two challenges faced by museums: the limitations of object handling and the desire for visitors to create their own interpretations. Using a Research-through-Design methodology, we built a performance-led Mixed Reality (MR) experience that lets museum visitors physically handle 3D prints or scans of museum objects to share personal stories about them. The stories are recorded and donated to the participating museum. We reflect on the complex design and the findings gained from a two-day in-the-wild deployment to explore engagement and disruption through manipulations of physicality, visuals, and scale; the transitions between spaces; and a trajectory of storytelling performance. We chart a wide scope for Performative Substitutional Reality and draw implications for VR, MR, and performance-led research in any context.2020JSJocelyn Spence et al.Digital Art Installations & Interactive PerformanceMuseum & Cultural Heritage DigitizationInteractive Narrative & Immersive StorytellingDIS
Encumbered Interaction: a Study of Musicians Preparing to PerformGuitars are physical instruments that require skillful two-handed use. Their use is also supported by diverse digital and physical resources, such as videos and chord charts. To understand the challenges of interacting with supporting resources at the same time as playing we conducted an ethnographic study of the preparation activities of working musicians. We observe successive stages of individual and collaborative preparation, in which working musicians engage with a diverse range of digital and physical resources to support their preparation. Interaction with this complex ecology of digital and physical resources is finely interwoven into their embodied musical practices, which are usually encumbered by having their instrument in hand, and often by playing. We identify challenges for augmenting guitars within the rehearsal process by supporting interaction that is encumbered, contextual and connected, and suggest a range of possible responses.2019JAJuan Pablo Martinez Avila et al.The University of NottinghamTeleoperated DrivingMusic Composition & Sound Design ToolsCHI
From Director's Cut to User's Cut: to Watch a Brain-Controlled Film is to Edit itIntroducing interactivity to films has proven a longstanding and difficult challenge due to their narrative-driven, linear and theatre-based nature. Previous research has suggested that Brain-Computer Interfaces (BCI) may be a promising approach but also revealed a tension between being immersed in the film and thinking about control. We report a performance-led and in-the-wild study of a BCI film called The MOMENT covering its design rationale and how it was experienced by the public as controllers, non-controllers and repeat viewers. Our findings suggest that BCI movies should be designed to be credibly controllable, generate personal versions, be watchable as linear films, encourage repeat viewing and fit the medium of cinema. They also reveal how viewers appreciated the sense of editing their own personal cuts, suggesting a new stance on introducing interactivity into lean-back media in which filmmakers release editorial control to users to make their own versions.2019RRRichard Ramchurn et al.The University of NottinghamBrain-Computer Interface (BCI) & NeurofeedbackInteractive Narrative & Immersive StorytellingCHI
Sensory Alignment in Immersive EntertainmentWhen we use digital systems to stimulate the senses, we typically stimulate only a subset of users' senses, leaving other senses stimulated by the physical world. This creates potential for misalignment between senses, where digital and physical stimulation give conflicting signals to users. We synthesize knowledge from HCI, traditional entertainments, and underlying sensory science research relating to how senses work when given conflicting signals. Using this knowledge we present a design dimension of sensory alignment, and show how this dimension presents opportunities for a range of creative strategies ranging from full alignment of sensory stimulation, up to extreme conflict between senses.2019JMJoe Marshall et al.University of NottinghamSocial & Collaborative VRImmersion & Presence ResearchGenerative AI (Text, Image, Music, Video)CHI
Seeing with New Eyes: Designing for In-the-Wild Museum GiftingThis paper presents the GIFT smartphone app, an artist-led Research through Design project benefitting from a three-day in-the-wild deployment. The app takes as its premise the generative potential of combining the contexts of gifting and museum visits. Visitors explore the museum, searching for objects that would most appeal to the gift-receiver they have in mind, then photographing those objects and adding audio messages for their receivers describing the motivation for their choices. This paper charts the designers' key aim of creating a new frame of mind using voice, and the most striking findings discovered during in-the-wild deployment in a museum -- 'seeing with new eyes' and fostering personal connections. We discuss empathy, motivation, and bottom-up personalisation in the productive space revealed by this combination of contexts. We suggest that this work reveals opportunities for designers of gifting services as well as those working in cultural heritage.2019JSJocelyn Spence et al.University of NottinghamMuseum & Cultural Heritage DigitizationInteractive Narrative & Immersive StorytellingCHI
Card Mapper: Enabling Data-Driven Reflections on Ideation CardsWe explore how usage data captured from ideation cards can enable reflection on design. We deployed a deck of ideation cards on a Masters level module over two years, developing the means to capture the students' designs into a digital repository. We created two visualisations to reveal the relative co-occurrences of the cards as concept space and the relative proximity of designs (through cards used in common) as design space. We used these to elicit reflections from the perspectives of students, teachers and card designers. Our findings inspire ideas for extending the data-driven use of ideation cards throughout the design process; informing the redesign of cards, the rules for using them and their live connection to supporting materials and enabling stakeholders to reflect and recognise challenges and opportunities. We also identified the need, and potential ways, to capture a richer design rationale, including annotations, discarded cards and varying card interpretations.2019DDDimitrios Darzentas et al.University of NottinghamInteractive Data VisualizationPrototyping & User TestingCHI
Abstract Machines: Overlaying Virtual Worlds on Physical RidesOverlaying virtual worlds onto existing physical rides and altering the sensations of motion can deliver new experiences of thrill, but designing how motion is mapped between physical ride and virtual world is challenging. In this paper, we present the notion of an abstract machine, a new form of intermediate design knowledge that communicates motion mappings at the level of metaphor, mechanism and implementation. Following a performance-led, in-the-wild approach we report lessons from creating and touring VR Playground, a ride that overlays four distinct abstract machines and virtual worlds on a playground swing. We compare the artist's rationale with riders' reported experiences and analysis of their physical behaviours to reveal the distinct thrills of each abstract machine. Finally, we discuss how to make and use abstract machines in terms of heuristics for designing motion mappings, principles for virtual world design and communicating experiences to riders.2019PTPaul Tennent et al.University of NottinghamImmersion & Presence ResearchInteractive Narrative & Immersive StorytellingDance & Body Movement ComputingCHI
The Performative Mirror SpaceInteractive mirrors, typically combining semi-transparent mirrors, digital screens and interaction mechanisms have been developed for a variety of application areas. Drawing on existing techniques to create interactive mirror spaces, we investigated their performative qualities through artistic discovery and collaborative prototyping. We document a linked set of design explorations and two public, site-specific experiences that brought together artists, communities, and HCI researchers. We illustrate the abstracted interactive mirror space that practitioners in the performance art, theatre and museum sectors can work with. In turn, we also discuss six performative design strategies concerning the use of physical context, movement and narrative that HCI researchers who wish to deploy interactive mirrors in more mainstream settings need to consider.2019RJRachel Jacobs et al.University of NottinghamDigital Art Installations & Interactive PerformanceInteractive Narrative & Immersive StorytellingCHI
Failing with Style: Designing for Aesthetic Failure in Interactive PerformanceFailure is a common artefact of challenging experiences, a fact of life for interactive systems but also a resource for aesthetic and improvisational performance. We present a study of how three professional pianists performed an interactive piano composition that included playing hidden codes within the music so as to control their path through the piece and trigger system actions. We reveal how apparent failures to play the codes occurred for diverse reasons including mistakes in their playing, limitations of the system, but also deliberate failures as a way of controlling the system, and how these failures provoked aesthetic and improvised responses from the performers. We propose that creative and performative interfaces should be designed to enable aesthetic failures and introduce a taxonomy that compares human approaches to failure with approaches to capable systems, revealing new creative design strategies of gaming, taming, riding and serving the system.2019AHAdrian Hazzard et al.University of NottinghamDigital Art Installations & Interactive PerformanceInteractive Narrative & Immersive StorytellingCHI
Designing the Audience Journey through Repeated ExperiencesWe report on the design, premiere and public evaluation of a multifaceted audience interface for a complex non-linear musical performance called Climb! which is particularly suited to being experienced more than once. This interface is designed to enable audiences to understand and appreciate the work, and integrates a physical instrument and staging, projected visuals, personal devices and an online archive. A public premiere concert comprising two performances of Climb! revealed how the audience reoriented to the second performance through growing understanding and comparison to the first. Using trajectories as an analytical framework for the audience ‘journey’ made apparent: how the trajectories of a single performance are embedded within the larger trajectories of a concert and the creative work as a whole; the distinctive demands of understanding and interpretation; and the potential of the archive in enabling appreciation across repeated performances.2018SBSteve Benford et al.The University of NottinghamDigital Art Installations & Interactive PerformanceInteractive Narrative & Immersive StorytellingCHI
Customizing Hybrid ProductsWe explore how the convergence of the digital and physical into hybrid products leads to new possibilities for customization. We report on a technology probe, a hybrid advent calendar with both paper form and digital layers of content, both of which were designed to be customizable. We reveal how over two hundred active users adapted its physical and digital aspects in various ways, some anticipated and familiar, but others surprising. This leads us to contribute concepts to help understand and design for hybrid customization – the idea of broad customization spanning physical and digital; end-to-end customization by different stakeholders along the value chain for a product; and the combination of these into customization maps.2018SBSteve Benford et al.University of Nottingham360° Video & Panoramic ContentUniversal & Inclusive DesignCustomizable & Personalized ObjectsCHI