Exploring the Experiences of Individuals Who are Blind or Low-Vision Using Object-Recognition Technologies in India Assistive technologies, such as smartphone-based object-recognition (OR) apps, provide visual assistance to people who are blind or low-vision to enable increased independent participation in society. While previous research has explored the functional accessibility of object-recognition technologies, little attention has been given to their social accessibility, particularly in interdependent socio-cultural contexts of the Global South. Through a mixed-methods approach, employing a seven-day diary study followed by one-on-one interviews with seven OR app users in India, we explore their experiences in depth. Our findings highlight the nuances of what interdependence looks like in a multicultural, Indian society, as people navigate public and private spheres with a camera-based assistive technology designed for independent, western contexts. We argue for the necessity to design assistive technologies following the interdependence framework that accommodates the social and cultural context of the Global South. Additionally, we propose design guidelines for assistive technologies in community-oriented societies, emphasizing community-centered approaches, cultural alignment, and locally adaptable designs.2025GIGesu India et al.Swansea UniversityVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Cognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)Universal & Inclusive DesignCHI
Cultivating Spoken Language Technologies for Unwritten LanguagesWe report on community-centered, collaborative research that weaves together HCI, natural language processing, linguistic, and design insights to develop spoken language technologies for unwritten languages. Across three visits to a Banjara farming community in India, we use participatory, technical, and creative methods to engage community members, collect spoken language photo annotations, and develop an information retrieval (IR) system. Drawing on orality theory, we interrogate assumptions and biases of current speech interfaces and create a simple application that leverages our IR system to match fluidly spoken queries with recorded annotations and surface corresponding photos. In-situ evaluations show how our novel approach returns reliable results and inspired the co-creation of media retrieval use-cases that are more appropriate in oral contexts. The very low (< 4h) spoken data requirements makes our approach adaptable to other contexts where languages are unwritten or have no digital language resources available.2024TRThomas Reitmaier et al.Swansea UniversityVoice User Interface (VUI) DesignIntelligent Voice Assistants (Alexa, Siri, etc.)Developing Countries & HCI for Development (HCI4D)CHI
Situating Automatic Speech Recognition Development within Communities of Under-heard Language SpeakersIn this paper we develop approaches to automatic speech recognition (ASR) development that suit the needs and functions of under-heard language speakers. Our novel contribution to HCI is to show how community-engagement can surface key technical and social issues and opportunities for more effective speech-based systems. We introduce a bespoke toolkit of technologies and showcase how we utilised the toolkit to engage communities of under-heard language speakers; and, through that engagement process, situate key aspects of ASR development in community contexts. The toolkit consists of (1) an information appliance to facilitate spoken-data collection on topics of community interest, (2) a mobile app to create crowdsourced transcripts of collected data, and (3) demonstrator systems to showcase ASR capabilities and to feed back research results to community members. Drawing on the sensibilities we cultivated through this research, we present a series of challenges to the orthodoxy of state-of-the-art approaches to ASR development.2023TRThomas Reitmaier et al.Swansea UniversityMultilingual & Cross-Cultural Voice InteractionDeveloping Countries & HCI for Development (HCI4D)User Research Methods (Interviews, Surveys, Observation)CHI
Opportunities and Challenges of Automatic Speech Recognition Systems for Low-Resource Language SpeakersAutomatic Speech Recognition (ASR) researchers are turning their attention towards supporting low-resource languages, such as isiXhosa or Marathi, with only limited training resources. We report and reflect on collaborative research across ASR & HCI to situate ASR-enabled technologies to suit the needs and functions of two communities of low-resource language speakers, on the outskirts of Cape Town, South Africa and in Mumbai, India. We build on longstanding community partnerships and draw on linguistics, media studies and HCI scholarship to guide our research. We demonstrate diverse design methods to: remotely engage participants; collect speech data to test ASR models; and ultimately field-test models with users. Reflecting on the research, we identify opportunities, challenges, and use-cases of ASR, in particular to support pervasive use of WhatsApp voice messaging. Finally, we uncover implications for collaborations across ASR & HCI that advance important discussions at CHI surrounding data, ethics, and AI.2022TRThomas Reitmaier et al.Swansea UniversityMultilingual & Cross-Cultural Voice InteractionExplainable AI (XAI)Cognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)CHI
Light in Light Out (LiLo) Displays: Harvesting and Manipulating Light to Provide Novel Forms of CommunicationMany of us daily encounter shadow and reflected light patterns alongside macro-level changes in ambient light levels. These are caused by elements - opaque objects, glass, mirrors, even clouds - in our environment interfacing with sunlight or artificial indoor lighting. Inspired by these phenomena, we explored ways of creating digitally-supported displays that use light, shade and reflection for output and harness the energy they need to operate from the sun or indoor ambient light. Through a set of design workshops we developed exemplar devices: SolarPix, ShadMo and GlowBoard. We detail their function and implementation, as well as evidencing their technical viability. The designs were informed by material understandings from the Global North and Global South and demonstrated in a cross-cultural workshop run in parallel in India and South Africa where community co-designers reflected on their uses and value given lived experience of their communication practices and unreliable energy networks.2022KSKrishna Seunarine et al.Swansea UniversitySustainable HCIEcological Design & Green ComputingDigital Art Installations & Interactive PerformanceCHI
Can't Touch This: Rethinking Public Technology in a COVID-19 EraWhat do pedestrian crossings, ATMs, elevators and ticket machines have in common? These are just a few of the ubiquitous yet essential elements of public-space infrastructure that rely on physical buttons or touchscreens; common interactions that, until recently, were considered perfectly safe to perform. This work investigates how we might integrate touchless technologies into public-space infrastructure in order to minimise physical interaction with shared devices in light of the ongoing COVID-19 pandemic. Drawing on an ethnographic exploration into how public utilities are being used, adapted or avoided, we developed and evaluated a suite of technology probes that can be either retro tted into, or replace, these services. In-situ community deployments of our probes demonstrate strong uptake and provide insight into how hands-free technologies can be adapted and utilised for the public domain; and, in turn, used to inform the future of walk-up-and use public technologies.2022JPJennifer Pearson et al.Swansea UniversityContext-Aware ComputingUbiquitous ComputingCHI
PV-Pix: Slum Community Co-design of Self-Powered Deformable Smart Messaging MaterialsWorking with emergent users in two of Mumbai’s slums, we explored the value and uses of photovoltaic (PV) self-powering digital materials. Through a series of co-design workshops, a diary study and responses by artists and craftspeople, we developed the PV-Pix concept for inter-home connections. Each PV-Pix element consists of a deformable energy harvesting material that, when actuated by a person in one home, changes its physical state both there and in a connected home. To explore the concept we considered two forms of PV-Pix: one uses rigid materials and the other flexible ones. We deployed two low-fidelity prototypes, each constructed of a grid of one PV-Pix type, in four slum homes over a four week period to further understand the usability and uses of the materials, eliciting interesting inter-family communication practices. Encouraged by these results we report on a first-step towards working prototypes and demonstrate the technical viability of the approach.2021DRDani Kalarikalayil Raju et al.Studio HasiShape-Changing Interfaces & Soft Robotic MaterialsParticipatory DesignSustainable HCICHI
Morphino: A Nature-Inspired Tool for the Design of Shape-Changing InterfacesThe HCI community has a strong and growing interest in shape-changing interfaces (SCIs) that can offer dynamic affordance. In this context, there is an increasing need for HCI researchers and designers to form close relationships with disciplines such as robotics and material science in order to be able to truly harness the state-of-the-art in morphing technologies. To help these synergies arise, we present \textit{Morphino}: a card-based toolkit to inspire shape-changing interface designs. Our cards bring together a collection of morphing mechanisms already established in the multidisciplinary literature and illustrate them through familiar examples from nature. We begin by detailing the design of the cards, based on a review of shape-change in nature; then, report on a series of design sessions conducted to demonstrate their usefulness in generating new ideas and in helping end-users gain a better understanding of the possibilities for shape-changing materials.2020IQIsabel P. S. Qamar et al.Shape-Changing Interfaces & Soft Robotic MaterialsDIS
PV-Tiles: Towards Closely-Coupled Photovoltaic and Digital Materials for Useful, Beautiful and Sustainable Interactive SurfacesThe interactive, digital future with its seductive vision of Internet-of-Things connected sensors, actuators and displays comes at a high cost in terms of both energy demands and the clutter it brings to the physical world. But what if such devices were made of materials that enabled them to self-power their interactive features? And, what if those materials were directly used to build aesthetically pleasing environments and objects that met practical physical needs as well as digital ones? In this paper we introduce PV-Tiles – a novel material that closely couples photovoltaic energy harvesting and light sensing materials with digital interface components. We consider potential contexts, use-cases and light gestures surfaced through co-creation workshops; and, present initial technological designs and prototypes. The work opens a new set of opportunities and collaborations between HCI and material science, stimulating technical and design pointers to accommodate and exploit the material's properties.2020YMYogesh Kumar Meena et al.Swansea UniversityShape-Changing Interfaces & Soft Robotic MaterialsSustainable HCIEcological Design & Green ComputingCHI
An Honest Conversation: Transparently Combining Machine and Human Speech Assistance in Public SpacesThere is widespread concern over the ways speech assistant providers currently use humans to listen to users' queries without their knowledge. We report two iterations of the TalkBack smart speaker, which transparently combines machine and human assistance. In the first, we created a prototype to investigate whether people would choose to forward their questions to a human answerer if the machine was unable to help. Longitudinal deployment revealed that most users would do so when given the explicit choice. In the second iteration we extended the prototype to draw upon spoken answers from previous deployments, combining machine efficiency with human richness. Deployment of this second iteration shows that this corpus can help provide relevant, human-created instant responses. We distil lessons learned for those developing conversational agents or other AI-infused systems about how to appropriately enlist human-in-the-loop information services to benefit users, task workers and system performance.2020TRThomas Reitmaier et al.Swansea UniversityIntelligent Voice Assistants (Alexa, Siri, etc.)Human-LLM CollaborationPrivacy by Design & User ControlCHI
StreetWise: Smart Speakers vs Human Help in Public Slum SettingsThis paper explores the use of conversational speech question and answer systems in the challenging context of public spaces in slums. A major part of this work is a comparison of the source and speed of the given responses; that is, either machine-powered and instant or human-powered and delayed. We examine these dimensions via a two-stage, multi-sited deployment. We report on a pilot deployment that helped refine the system, and a second deployment involving the installation of nine of each type of system within a large Mumbai slum for a 40-day period, resulting in over 12,000 queries. We present the findings from a detailed analysis and comparison of the two question-answer corpora; discuss how these insights might help improve machine-powered smart speakers; and, highlight the potential benefits of multi-sited public speech installations within slum environments.2019JPJennifer Pearson et al.Swansea UniversityIntelligent Voice Assistants (Alexa, Siri, etc.)Developing Countries & HCI for Development (HCI4D)CHI
Sustainabot – Exploring the Use of Everyday Foodstuffs as Output and Input for and with Emergent UsersMainstream digital interactions are spread over a plethora of devices and form-factors, from mobiles to laptops; printouts to large screens. For emergent users, however, such abundance of choice is rarely accessible or affordable. In particular, viewing mobile content on a larger screen, or printing out copies, is often not available. In this paper we present Sustainabot – a small robot printer that uses everyday materials to print shapes and patterns from mobile phones. Sustainabot was proposed and developed by and with emergent users through a series of co-creation workshops. We begin by discussing this process, then detail the open-source mobile printer prototype. We carried out two evaluations of Sustainabot, the first focused on printing with materials in situ, and the second on understandability of its output. We present these results, and discuss opportunities and challenges for similar developments. We conclude by highlighting where and how similar devices could be used in future.2019SRSimon Robinson et al.Swansea UniversityUbiquitous ComputingCustomizable & Personalized ObjectsCHI
PickCells: A Physically Reconfigurable Cell-composed TouchscreenTouchscreens are the predominant medium for interactions with digital services; however, their current fixed form factor narrows the scope for rich physical interactions by limiting interaction possibilities to a single, planar surface. In this paper we introduce the concept of PickCells, a fully re-configurable device concept composed of cells, that breaks the mould of rigid screens and explores a modular system that affords rich sets of tangible interactions and novel across-device relationships. Through a series of co-design activities -- involving HCI experts and potential end-users of such systems -- we synthesised a design space aimed at inspiring future research, giving researchers and designers a framework in which to explore modular screen interactions. The design space we propose unifies existing works on modular touch surfaces under a general framework and broadens horizons by opening up unexplored spaces providing new interaction possibilities. In this paper, we present the PickCells concept, a design space of modular touch surfaces, and propose a toolkit for quick scenario prototyping.2019AGAlix Goguey et al.Swansea UniversityShape-Changing Interfaces & Soft Robotic MaterialsCustomizable & Personalized ObjectsCHI
Pulp Friction: Exploring the Finger Pad Periphery for Subtle Haptic FeedbackCurrent haptic feedback techniques on handheld devices are applied to the finger pad or the palm of the user. These state-of-the-art approaches are coarse-grained and tend to be intrusive, rather than subtle. In contrast, we present a new feedback technique that applies stimuli around the periphery of the finger pulp, demonstrating how this can provide rich, nuanced haptic information. We use a reconfigurable haptic device employing a ferromagnetic marble for back-of-the device handheld use, which, for the first time, probes, without instrumenting the user, the periphery of the distal phalanx with localised stimulation. We present the design-space afforded by this new technique and evaluate the human-factors of finger-peripheral touch interaction in a controlled user-study. We report results with marbles of different diameters, speeds and a combination of poking, lateral vibration and patterns; present the resulting design guidelines for finger-periphery haptic feedback; and, illustrate its potential with use case scenarios.2019AGAlix Goguey et al.Swansea UniversityVibrotactile Feedback & Skin StimulationCHI
Mobile UX – The Next Ten Years?Mobile phones are the ubiquitous platform used by billions of people globally, every day. However, two concerns signal a pause for reflection and change. First, while mobiles have rapidly become indispensable, the effect that constant device use has on our lives, our experiences, and the interactions we have with others, has caused growing discomfort. At the same time, there is a broad sense that mainstream mobile devices have fallen into a period of innovation limbo, with recent releases seemingly being distinguished only by ever narrowing feature gaps. As a recent Economist article bleakly reports, ‘‘More black rectangles made their debut”. This course will challenge attendees to play a part in reinvigorating mobile interaction design. We celebrate the success that is apps, services, and the hugely popular ecology of mobile devices, but want to promote a return to radical innovation. We have been fortunate enough to have collaborated with a broad range of industrial and academic researchers and practitioners over many years. More importantly, however, we have worked with a wide range of people who are not considered to be typical “future makers,” and are also not usually considered when designing mobile user experiences. Typically these people—who have been called “emergent” users—are often drawn from developing regions, with lower literacy, lower socioeconomic conditions, and other constraints. Our experience in working with these people has demonstrated how their unique and contrasting outlooks on both technology and the world and ways of seeing it are invaluable in generating radically new and exciting digital innovations.2018SRSimon Robinson et al.Swansea UniversityDeveloping Countries & HCI for Development (HCI4D)Participatory DesignSustainable HCICHI
Mobile UX – The Next Ten Years?Mobile phones are the ubiquitous platform used by billions of people globally, every day. However, two concerns signal a pause for reflection and change. First, while mobiles have rapidly become indispensable, the effect that constant device use has on our lives, our experiences, and the interactions we have with others, has caused growing discomfort. At the same time, there is a broad sense that mainstream mobile devices have fallen into a period of innovation limbo, with recent releases seemingly being distinguished only by ever narrowing feature gaps. As a recent Economist article bleakly reports, ‘‘More black rectangles made their debut”. This course will challenge attendees to play a part in reinvigorating mobile interaction design. We celebrate the success that is apps, services, and the hugely popular ecology of mobile devices, but want to promote a return to radical innovation. We have been fortunate enough to have collaborated with a broad range of industrial and academic researchers and practitioners over many years. More importantly, however, we have worked with a wide range of people who are not considered to be typical “future makers,” and are also not usually considered when designing mobile user experiences. Typically these people—who have been called “emergent” users—are often drawn from developing regions, with lower literacy, lower socioeconomic conditions, and other constraints. Our experience in working with these people has demonstrated how their unique and contrasting outlooks on both technology and the world and ways of seeing it are invaluable in generating radically new and exciting digital innovations.2018SRSimon Robinson et al.Swansea UniversityContext-Aware ComputingDeveloping Countries & HCI for Development (HCI4D)CHI
Make Yourself at Phone: Reimagining Mobile Interaction Architectures With Emergent UsersWe present APPropriate – a novel mobile design to allow users to temporarily annex any Android device for their own use. APPropriate is a small, cheap storage pod, designed to be easily carried in a pocket or hidden within clothing. Its purpose is simple: to hold a copy of the local content an owner has on their mobile, liberating them from carrying a phone, or allowing them to use another device that provides advantages over their own. Picking up another device when carrying APPropriate transfers all pertinent content to the borrowed device (using local no-cost WiFi from the APPropriate device), transforming it to give the impression that they are using their own phone. While APPropriate is useful for a wide range of contexts, the design was envisaged through a co-design process with resource-constrained emergent users in three countries. Lab studies and a subsequent deployment on participants’ own devices identified key benefits of the approach in these contexts, including for security, resource sharing, and privacy.2018SRSimon Robinson et al.Swansea UniversityPrivacy Perception & Decision-MakingUbiquitous ComputingCHI
Revisiting “Hole in the Wall” Computing: Private Smart Speakers and Public Slum SettingsMillions of homes worldwide enjoy access to digital content and services through smart speakers such as Amazon’s Echo and Google’s Home. Promotional materials and users’ own videos typically show homes that have many well-resourced rooms, with good power and data infrastructures. Over the last several years, we have been working with slum communities in India, whose dwellings are usually very compact (one or two rooms), personal home WiFi is almost unheard of, power infrastructures are far less robust, and financial resources put such smart speakers out of individual household reach. Inspired by the “hole in the wall” internet-kiosk programme, we carried out workshops with slum inhabitants to uncover issues and opportunities for providing a smart-speaker-type device in public areas and passageways. We designed and deployed a simple probe that allowed passers-by to ask and receive answers to questions. In this paper, we present the findings of this work, and a design space for such devices in these settings.2018SRSimon Robinson et al.Swansea UniversityVoice User Interface (VUI) DesignDeveloping Countries & HCI for Development (HCI4D)CHI
Tangible Drops: A Visio-Tactile Display Using Actuated Liquid-Metal DropletsWe present Tangible Drops, a visio-tactile display that for the first time provides physical visualization and tactile feedback using a planar liquid interface. It presents digital information interactively by tracing dynamic patterns on horizontal flat surfaces using liquid metal drops on a programmable electrode array. It provides tactile feedback with directional information in the 2D vector plane using linear locomotion and/or vibration of the liquid metal drops. We demonstrate move, oscillate, merge, split and dispense-from-reservoir functions of the liquid metal drops by consuming low power (450 mW per electrode) and low voltage (8--15 V). We report on results of our empirical study with 12 participants on tactile feedback using 8 mm diameter drops, which indicate that Tangible Drops can convey tactile sensations such as changing speed, varying direction and controlled oscillation with no visual feedback. We present the design space and demonstrate the applications of Tangible Drops, and conclude by suggesting potential future applications for the technique.2018DSDeepak Ranjan Sahoo et al.Swansea UniversityVibrotactile Feedback & Skin StimulationData PhysicalizationShape-Changing Materials & 4D PrintingCHI