XAIR: A Framework of Explainable AI in Augmented RealityExplainable AI (XAI) has established itself as an important component of AI-driven interactive systems. With Augmented Reality (AR) becoming more integrated in daily lives, the role of XAI also becomes essential in AR because end-users will frequently interact with intelligent services. However, it is unclear how to design effective XAI experiences for AR. We propose XAIR, a design framework that addresses when, what, and how to provide explanations of AI output in AR. The framework was based on a multi-disciplinary literature review of XAI and HCI research, a large-scale survey probing 500+ end-users’ preferences for AR-based explanations, and three workshops with 12 experts collecting their insights about XAI design in AR. XAIR's utility and effectiveness was verified via a study with 10 designers and another study with 12 end-users. XAIR can provide guidelines for designers, inspiring them to identify new design opportunities and achieve effective XAI designs in AR.2023XXXuhai Xu et al.Reality Labs Research, University of WashingtonAR Navigation & Context AwarenessExplainable AI (XAI)CHI
NFCStack: Identifiable Physical Building Blocks that Support Concurrent Construction and Frictionless InteractionIn this paper, we propose NFCStack, which is a physical building block system that supports stacking and frictionless interaction and is based on near-field communication (NFC). This system consists of a portable station that can support and resolve the order of three types of passive identifiable stackable: bricks, boxes, and adapters. The bricks support stable and sturdy physical construction, whereas the boxes support frictionless tangible interactions. The adapters provide an interface between the aforementioned two types of stackable and convert the top of a stack into a terminal for detecting interactions between NFC-tagged objects. In contrast to existing systems based on NFC or radio-frequency identification technologies, NFCStack is portable, supports simultaneous interactions, and resolves stacking and interaction events responsively, even when objects are not strictly aligned. Evaluation results indicate that the proposed system effectively supports 12 layers of rich-ID stacking with the three types of building block, even if every box is stacked with a 6-mm offset. The results also indicate possible generalized applications of the proposed system, including 2.5-dimensional construction. The interaction styles are described using several educational application examples, and the design implications of this research are explained.2022CLChi-Jung Lee et al.Aging-Friendly Technology DesignCircuit Making & Hardware PrototypingUIST
AccessibleCircuits: Adaptive Add-On Circuit Components for People with Blindness or Low VisionIn this paper, we propose the designs for low cost and 3D-printable add-on components to adapt existing breadboards, circuit components and electronics tools for users with blind or low vision (BLV). Through an initial user study, we identified several barriers to entry for beginners with BLV in electronics and circuit prototyping. These barriers guided the design and development of our add-on components. We focused on developing adaptations that provide additional information about the specific component pins and breadboard holes, modify tools to make them easier to use for users with BLV, and expand non-visual feedback (e.g., audio, tactile) for tasks that require vision. Through a second user study, we demonstrated that our adaptations can effectively overcome the accessibility barriers in breadboard circuit prototyping.2021RCRuei-Che Chang et al.Dartmouth CollegeMotor Impairment Assistive Input TechnologiesCircuit Making & Hardware PrototypingCHI
Project Tasca : Enabling Touch and Contextual Interactions with a Pocket-based Textile SensorWe present Project Tasca, a pocket-based textile sensor that detects user input and recognizes everyday objects that a user carries in the pockets of a pair of pants (e.g., keys, coins, electronic devices, or plastic items). By creating a new fabric-based sensor capable of detecting in-pocket touch and pressure, and recognizing metallic, non-metallic, and tagged objects inside the pocket, we enable a rich variety of subtle, eyes-free, and always-available input, as well as context-driven interactions in wearable scenarios. We developed our prototype by integrating four distinct types of sensing methods, namely: inductive sensing, capacitive sensing, resistive sensing, and NFC in a multi-layer fabric structure into the form factor of a jeans pocket. Through a ten-participant study, we evaluated the performance of our prototype across 11 common objects including hands, 8 force gestures, and 30 NFC tag placements. We yielded 92.3% personal cross-validation accuracy for object recognition, 96.4% accuracy for gesture recognition, and 100% accuracy for detecting NFC tags at close distance . We concluded by demonstrating the interactions enabled by our pocket-based sensor in several applications.2021TWTe-Yen Wu et al.Microsoft Research, Dartmouth CollegeHaptic WearablesFoot & Wrist InteractionContext-Aware ComputingCHI
Capacitivo: Contact-Based Object Recognition on Interactive Fabrics using Capacitive Sensing We present Capacitivo, a contact-based object recognition technique developed for interactive fabrics, using capacitive sensing. Unlike prior work that has focused on metallic objects, our technique recognizes non-metallic objects such as food, different types of fruits, liquids, and other types of objects that are often found around a home or in a workplace. To demonstrate our technique, we created a prototype composed of a 12 x 12 grid of electrodes, made from conductive fabric attached to a textile substrate. We designed the size and separation between the electrodes to maximize the sensing area and sensitivity. We then used a 10-person study to evaluate the performance of our sensing technique using 20 different objects, which yielded a 94.5% accuracy rate. We conclude this work by presenting several different application scenarios to demonstrate unique interactions that are enabled by our technique on fabrics.2020TWTe-Yen Wu et al.Haptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsCircuit Making & Hardware PrototypingUIST
Fabriccio: Touchless Gestural Input on Interactive FabricsWe present Fabriccio, a touchless gesture sensing technique developed for interactive fabrics using Doppler motion sensing. Our prototype was developed using a pair of loop antennas (one for transmitting and the other for receiving), made of conductive thread that was sewn onto a fabric substrate. The antenna type, configuration, transmission lines, and operating frequency were carefully chosen to balance the complexity of the fabrication process and the sensitivity of our system for touchless hand gestures, performed at a 10 cm distance. Through a ten-participant study, we evaluated the performance of our proposed sensing technique across 11 touchless gestures as well as 1 touch gesture. The study result yielded a 92.8% cross-validation accuracy and 85.2% leave-one-session-out accuracy. We conclude by presenting several applications to demonstrate the unique interactions enabled by our technique on soft objects.2020TWTe-Yen Wu et al.Dartmouth CollegeHaptic WearablesHand Gesture RecognitionElectronic Textiles (E-textiles)CHI
ThreadSense: Locating Touch on an Extremely Thin Interactive ThreadWe propose a new sensing technique for one-dimensional touch input workable on an interactive thread of less than 0.4 mm thick. Our technique locates up to two touches using impedance sensing with a spacing resolution unachievable by the existing methods. Our approach is also unique in that it locates a touch based on a mathematical model describing the change in thread impedance in relation to the touch locations. This allows the system to be easily calibrated by the user touching a known location(s) on the thread. The system can thus quickly adapt to various environmental settings and users. A system evaluation showed that our system could track the slide motion of a finger with an average error distance of 6.13 mm and 4.16 mm using one and five touches for calibration, respectively. The system could also distinguish between single touch and two concurrent touches with an accuracy of 99% and could track two concurrent touches with an average error distance of 8.55 mm. We demonstrate new interactions enabled by our sensing approach in several unique applications.2020PKPin-sung Ku et al.Dartmouth College; National Taiwan UniversityHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsCHI
BiTipText: Bimanual Eyes-Free Text Entry on a Fingertip KeyboardWe present a bimanual text input method on a miniature fingertip keyboard, that invisibly resides on the first segment of a user's index finger on both hands. Text entry can be carried out using the thumb-tip to tap the tip of the index finger. The design of our keyboard layout followed an iterative process, where we first conducted a study to understand the natural expectation of the handedness of the keys in a QWERTY layout for users. Among a choice of 67,108,864 design variations, we identified 1295 candidates offering a good satisfaction for user expectations. Based on these results, we computed an optimized bimanual keyboard layout, while considering the joint optimization problems of word ambiguity and movement time. Our user evaluation revealed that participants achieved an average text entry speed of 23.4 WPM.2020ZXZheer Xu et al.Dartmouth CollegeFoot & Wrist InteractionMotor Impairment Assistive Input TechnologiesCHI
TangibleCircuits: An Interactive 3D Printed Circuit Education Tool for People with Visual ImpairmentsWe present a novel haptic and audio feedback device that allows blind and visually impaired (BVI) users to understand circuit diagrams. TangibleCircuits allows users to interact with a 3D printed tangible model of a circuit which provides audio tutorial directions while being touched. Our system comprises an automated parsing algorithm which extracts 3D printable models as well as an audio interfaces from a Fritzing diagram. To better understand the requirements of designing technology to assist BVI users in learning hardware computing, we conducted a series of formative inquiries into the accessibility limitations of current circuit tutorial technologies. In addition, we derived insights and design considerations gleaned from conducting a formal comparative user study to understand the effectiveness of TangibleCircuits as a tutorial system. We found that BVI users were better able to understand the geometric, spatial and structural circuit information using TangibleCircuits, as well as enjoyed learning with our tool.2020JDJosh Urban Davis et al.Dartmouth CollegeElectrical Muscle Stimulation (EMS)Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Special Education TechnologyCHI
Zippro: The Design and Implementation of An Interactive ZipperZippers are common in a wide variety of objects that we use daily. This work investigates how we can take advantage of such common daily activities to support seamless interaction with technology. We look beyond simple zipper-sliding interactions explored previously to determine how to weave foreground and background interactions into a vocabulary of natural usage patterns. We begin by conducting two user studies to understand how people typically interact with zippers. The findings identify several opportunities for zipper input and sensing, which inform the design of Zippro, a self-contained prototype zipper slider, which we evaluate with a standard jacket zipper. We conclude by demonstrating several applications that make use of the identified foreground and background input methods.2020PKPin-sung Ku et al.Dartmouth College & National Taiwan UniversityShape-Changing Interfaces & Soft Robotic MaterialsCHI
Proxino: Enabling Prototyping of Virtual Circuits With Physical ProxiesWe propose blending the virtual and physical worlds for prototyping circuits using physical proxies. With physical proxies, real-world components (e.g. a motor, or light sensor) can be used with a virtual counterpart for a circuit designed in software. We demonstrate this concept in Proxino, and elucidate the new scenarios it enables for makers, such as remote collaboration with physically distributed electronics components. We compared our hybrid system and its output with designs of real circuits to determine the difference through a system evaluation and observed minimal differences. We then present the results of an informal study with 9 users, where we gathered feedback on the effectiveness of our system in different working conditions (with a desktop, using a mobile, and with a remote collaborator). We conclude by sharing our lessons learned from our system and discuss directions for future research that blend physical and virtual prototyping for electronic circuits.2019TWTe-Yen Wu et al.Desktop 3D Printing & Personal FabricationCircuit Making & Hardware PrototypingUIST
TipText: Eyes-Free Text Entry on a Fingertip KeyboardIn this paper, we propose and investigate a new text entry technique using micro thumb-tip gestures. Our technique features a miniature QWERTY keyboard residing invisibly on the first segment of the user’s index finger. Text entry can be carried out using the thumb-tip to tap the tip of the index finger. The keyboard layout was optimized for eyes-free input by utilizing a spatial model reflecting the users’ natural spatial awareness of key locations on the index finger. We present our approach of designing and optimizing the keyboard layout through a series of user studies and computer simulated text entry tests over 1,146,484 possibilities in the design space. The outcome is a 2×3 grid with the letters highly confining to the alphabetic and spatial arrangement of QWERTY. Our one-day user evaluation showed that participants achieved an average text entry speed of 11.9 WPM and were able to type as fast as 13.3 WPM towards the end of the experiment.2019ZXZheer Xu et al.Foot & Wrist InteractionMotor Impairment Assistive Input TechnologiesUIST