Using an Array of Needles to Create Solid Knitted ShapesTextiles offer many advantages as a fabrication material, particularly when viewed from the perspective of human use. Until recently most textile fabrication processes were limited to the creation of surface-based forms. Prior work on Solid Knitting demonstrated that it is possible to create some solid knitted forms. We present a different machine design using a 2D bed of knitting needles to fabricate additional solid knitted forms. This approach provides substantially more flexibility on how to structure stitches as yarn paths inside a volume. We describe a small prototype 6x6 needle machine, and demonstrate that it can create traditional knits, horizontal knits (knitted in the plane of the needles), and solid knits, including overhangs, and pyramidal forms. We conclude by considering future directions and the current limitation of this proof-of-concept design and how it holds the promise of creating knitted objects with engineered stiffness, elasticity, and density properties throughout their volume.2025FGFrancois Guimbretiere et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingUIST
SplatOverflow: Asynchronous Hardware TroubleshootingAs tools for designing and manufacturing hardware become more accessible, smaller producers can develop and distribute novel hardware. However, processes for supporting end-user hardware troubleshooting or routine maintenance aren't well defined. As a result, providing technical support for hardware remains ad-hoc and challenging to scale. Inspired by patterns that helped scale software troubleshooting, we propose a workflow for asynchronous hardware troubleshooting: SplatOverflow. SplatOverflow creates a novel boundary object, the SplatOverflow scene, that users reference to communicate about hardware. A scene comprises a 3D Gaussian Splat of the user's hardware registered onto the hardware’s CAD model. The splat captures the current state of the hardware, and the registered CAD model acts as a referential anchor for troubleshooting instructions. With SplatOverflow, remote maintainers can directly address issues and author instructions in the user’s workspace. Workflows containing multiple instructions can easily be shared between users and recontextualized in new environments. In this paper, we describe the design of SplatOverflow, the workflows it enables, and its utility to different kinds of users. We also validate that non-experts can use SplatOverflow to troubleshoot common problems with a 3D printer in a usability study. Project Page: https://amritkwatra.com/research/splatoverflow.2025AKAmritansh Kwatra et al.Cornell Tech, Information ScienceCrowdsourcing Task Design & Quality ControlCircuit Making & Hardware PrototypingPrototyping & User TestingCHI
EchoGuide: Active Acoustic Guidance for LLM-Based Eating Event Analysis from Egocentric VideosSelf-recording eating behaviors is a step towards a healthy lifestyle recommended by many health professionals. However, the current practice of manually recording eating activities using paper records or smartphone apps is often unsustainable and inaccurate. Smart glasses have emerged as a promising wearable form factor for tracking eating behaviors, but existing systems primarily identify when eating occurs without capturing details of the eating activities (E.g., what is being eaten). In this paper, we present EchoGuide, an application and system pipeline that leverages low-power active acoustic sensing to guide head-mounted cameras to capture egocentric videos, enabling efficient and detailed analysis of eating activities. By combining active acoustic sensing for eating detection with video captioning models and large-scale language models for retrieval augmentation, EchoGuide intelligently clips and analyzes videos to create concise, relevant activity records on eating. We evaluated EchoGuide with 9 participants in naturalistic settings involving eating activities, demonstrating high-quality summarization and significant reductions in video data needed, paving the way for practical, scalable eating activity tracking.2024VPVineet Parikh et al.Diet Tracking & Nutrition ManagementSleep & Stress MonitoringBiosensors & Physiological MonitoringUbiComp
MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on EyeglassesWe introduce MunchSonic, an AI-powered active acoustic sensing system integrated into eyeglasses to track fine-grained dietary actions. MunchSonic emits inaudible ultrasonic waves from the eyeglass frame, with the reflected signals capturing detailed positions and movements of body parts, including the mouth, jaw, arms, and hands involved in eating. These signals are processed by a deep learning pipeline to classify six actions: hand-to-mouth movements for food intake, chewing, drinking, talking, face-hand touching, and other activities (null). In an unconstrained study with 12 participants, MunchSonic achieved a 93.5% macro F1-score in a user-independent evaluation with a 2-second resolution in tracking these actions, also demonstrating its effectiveness in tracking eating episodes and food intake frequency within those episodes.2024SMSaif Mahmud et al.Diet Tracking & Nutrition ManagementBiosensors & Physiological MonitoringUbiComp
SeamPose: Repurposing Seams as Capacitive Sensors in a Shirt for Upper-Body Pose TrackingSeams are areas of overlapping fabric formed by stitching two or more pieces of fabric together in the cut-and-sew apparel manufacturing process. In SeamPose, we repurposed seams as capacitive sensors in a shirt for continuous upper-body pose estimation. Compared to previous all-textile motion-capturing garments that place the electrodes on the clothing surface, our solution leverages existing seams inside of a shirt by machine-sewing insulated conductive threads over the seams. The unique invisibilities and placements of the seams afford the sensing shirt to look and wear similarly as a conventional shirt while providing exciting pose-tracking capabilities. To validate this approach, we implemented a proof-of-concept untethered shirt with 8 capacitive sensing seams. With a 12-participant user study, our customized deep-learning pipeline accurately estimates the relative (to the pelvis) upper-body 3D joint positions with a mean per joint position error (MPJPE) of 6.0 cm. SeamPose represents a step towards unobtrusive integration of smart clothing for everyday pose estimation.2024TYTianhong Catherine Yu et al.Haptic WearablesHuman Pose & Activity RecognitionBiosensors & Physiological MonitoringUIST
EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a WristbandOur hands serve as a fundamental means of interaction with the world around us. Therefore, understanding hand poses and interaction contexts is critical for human-computer interaction (HCI). We present EchoWrist, a low-power wristband that continuously estimates 3D hand poses and recognizes hand-object interactions using active acoustic sensing. EchoWrist is equipped with two speakers emitting inaudible sound waves toward the hand. These sound waves interact with the hand and its surroundings through reflections and diffractions, carrying rich information about the hand's shape and the objects it interacts with. The information captured by the two microphones goes through a deep learning inference system that recovers hand poses and identifies various everyday hand activities. Results from the two 12-participant user studies show that EchoWrist is effective and efficient at tracking 3D hand poses and recognizing hand-object interactions. Operating at 57.9 mW, EchoWrist can continuously reconstruct 20 3D hand joints with MJEDE of 4.81 mm and recognize 12 naturalistic hand-object interactions with 97.6% accuracy.2024CLChi-Jung Lee et al.Cornell UniversityHand Gesture RecognitionFoot & Wrist InteractionCHI
EyeEcho: Continuous and Low-power Facial Expression Tracking on GlassesIn this paper, we introduce EyeEcho, a minimally-obtrusive acoustic sensing system designed to enable glasses to continuously monitor facial expressions. It utilizes two pairs of speakers and microphones mounted on glasses, to emit encoded inaudible acoustic signals directed towards the face, capturing subtle skin deformations associated with facial expressions. The reflected signals are processed through a customized machine-learning pipeline to estimate full facial movements. EyeEcho samples at 83.3 Hz with a relatively low power consumption of 167 mW. Our user study involving 12 participants demonstrates that, with just four minutes of training data, EyeEcho achieves highly accurate tracking performance across different real-world scenarios, including sitting, walking, and after remounting the devices. Additionally, a semi-in-the-wild study involving 10 participants further validates EyeEcho's performance in naturalistic scenarios while participants engage in various daily activities. Finally, we showcase EyeEcho's potential to be deployed on a commercial-off-the-shelf (COTS) smartphone, offering real-time facial expression tracking.2024KLKe Li et al.Cornell UniversityHand Gesture RecognitionEye Tracking & Gaze InteractionHuman Pose & Activity RecognitionCHI
VRoxy: Wide-Area Collaboration From an Office Using a VR-Driven Robotic ProxyRecent research in robotic proxies has demonstrated that one can automatically reproduce many non-verbal cues important in co-located collaboration. However, they often require a symmetrical hardware setup in each location. We present the VRoxy system, designed to enable access to remote spaces through a robotic embodiment, using a VR headset in a much smaller space, such as a personal office. VRoxy maps small movements in VR space to larger movements in the physical space of the robot, allowing the user to navigate large physical spaces easily. Using VRoxy, the VR user can quickly explore and navigate in a low-fidelity rendering of the remote space. Upon the robot's arrival, the system uses the feed of a 360 camera to support real-time interactions. The system also facilitates various interaction modalities by rendering the micro-mobility around shared spaces, head and facial animations, and pointing gestures on the proxy. We demonstrate how our system can accommodate mapping multiple physical locations onto a unified virtual space. In a formative study, users could complete a design decision task where they navigated and collaborated in a complex 7.5m x 5m layout using a 3m x 2m VR space.2023MSMose Sakashita et al.Teleoperation & TelepresenceUIST
HPSpeech: Silent Speech Interface for Commodity HeadphonesWe present HPSpeech, a silent speech interface for commodity headphones. HPSpeech utilizes the existing speakers of the headphones to emit inaudible acoustic signals. The movements of the temporomandibular joint (TMJ) during speech modify the reflection pattern of these signals, which are captured by a microphone positioned inside the headphones. To evaluate the performance of HPSpeech, we tested it on two headphones with a total of 18 participants. The results demonstrated that HPSpeech successfully recognized 8 popular silent speech commands for controlling the music player with an accuracy over 90%. While our tests use modified commodity hardware (both with and without active noise cancellation), our results show that sensing the movement of the TMJ could be as simple as a firmware update for ANC headsets which already include a microphone inside the hear cup. This leaves us to believe that this technique has great potential for rapid deployment in the near future. We further discuss the challenges that need to be addressed before deploying HPSpeech at scale.2023RZRuidong Zhang et al.Voice User Interface (VUI) DesignUbiComp
EchoNose: Sensing Mouth, Breathing and Tongue Gestures inside Oral Cavity using a Non-contact Nose InterfaceSensing movements and gestures inside the oral cavity has been a long-standing challenge for the wearable research community. This paper introduces EchoNose, a novel nose interface that explores a unique sensing approach to recognize gestures related to mouth, breathing, and tongue by analyzing the acoustic signal reflections inside the nasal and oral cavities. The interface incorporates a speaker and a microphone placed at the nostrils, emitting inaudible acoustic signals and capturing the corresponding reflections. These received signals were processed using a customized data processing and machine learning pipeline, enabling the distinction of 16 gestures involving speech, tongue, and breathing. A user study with 10 participants demonstrates that EchoNose achieves an average accuracy of 93.7% in recognizing these 16 gestures. Based on these promising results, we discuss the potential opportunities and challenges associated with applying this innovative nose interface in various future applications.2023RSRujia Sun et al.Electrical Muscle Stimulation (EMS)Hand Gesture RecognitionBrain-Computer Interface (BCI) & NeurofeedbackUbiComp
EchoSpeech: Continuous Silent Speech Recognition on Minimally-obtrusive Eyewear Powered by Acoustic SensingWe present EchoSpeech, a minimally-obtrusive silent speech interface (SSI) powered by low-power active acoustic sensing. EchoSpeech uses speakers and microphones mounted on a glass-frame and emits inaudible sound waves towards the skin. By analyzing echos from multiple paths, EchoSpeech captures subtle skin deformations caused by silent utterances and uses them to infer silent speech. With a user study of 12 participants, we demonstrate that EchoSpeech can recognize 31 isolated commands and 3-6 figure connected digits with 4.5% (std 3.5%) and 6.1% (std 4.2%) Word Error Rate (WER), respectively. We further evaluated EchoSpeech under scenarios including walking and noise injection to test its robustness. We then demonstrated using EchoSpeech in demo applications in real-time operating at 73.3mW, where the real-time pipeline was implemented on a smartphone with only 1-6 minutes of training data. We believe that EchoSpeech takes a solid step towards minimally-obtrusive wearable SSI for real-life deployment.2023RZRuidong Zhang et al.Cornell UniversityVibrotactile Feedback & Skin StimulationVoice User Interface (VUI) DesignBiosensors & Physiological MonitoringCHI
ReMotion: Supporting Remote Collaboration in Open Space with Automatic Robotic EmbodimentDesign activities, such as brainstorming or critique, often take place in open spaces combining whiteboards and tables to present artefacts. In co-located settings, peripheral awareness enables participants to understand each other’s locus of attention with ease. However, these spatial cues are mostly lost while using videoconferencing tools. Telepresence robots could bring back a sense of presence, but controlling them is distracting. To address this problem, we present ReMotion, a fully automatic robotic proxy designed to explore a new way of supporting non-collocated open-space design activities. ReMotion combines a commodity body tracker (Kinect) to capture a user’s location and orientation over a wide area with a minimally invasive wearable system (NeckFace) to capture facial expressions. Due to its omnidirectional platform, ReMotion embodiment can render a wide range of body movements. A formative evaluation indicated that our system enhances the sharing of attention and the sense of co-presence enabling seamless movement-in-space during a design review task.2023MSMose Sakashita et al.Cornell UniversityHuman-Robot Collaboration (HRC)Teleoperation & TelepresenceCHI
RemoteCoDe: Robotic Embodiment for Enhancing Peripheral Awareness in a Remote Collaborative TaskMany collaborative design activities are centered around a shared artifact such as a low fidelity model of a building, or a circuit implemented on a breadboard. In such settings, collocation is extremely valuable as participants can easily infer each other’s focus of attention through peripheral awareness. These cues are often lost in traditional video conferencing systems such as Zoom. In this paper, we present an embodied remote presence system designed to support design activities that involve a physical artifact by enhancing peripheral awareness. In our system, a remote user's focus of attention is tracked by an iPhone's TruthDepth camera and rendered locally using an articulated display. Our implementation can track the wide range of head movements that occur when one switches attention between a physical artifact, a laptop, and their collaborator. To support possible discrepency between local and remote layout, head movements are remapped as robotic movements to correspond to these key elements in the local user's space. We report on the results of an evaluation characterizing the system's remapping accuracy and its ability to support peripheral awareness of a remote participant's locus of attention.2022MSMose Sakashita et al.Remote and Hybrid CollaborationsCSCW
RemoteCoDe: Robotic Embodiment for Enhancing Peripheral Awareness in a Remote Collaborative TaskMany collaborative design activities are centered around a shared artifact such as a low fidelity model of a building, or a circuit implemented on a breadboard. In such settings, collocation is extremely valuable as participants can easily infer each other’s focus of attention through peripheral awareness. These cues are often lost in traditional video conferencing systems such as Zoom. In this paper, we present an embodied remote presence system designed to support design activities that involve a physical artifact by enhancing peripheral awareness. In our system, a remote user's focus of attention is tracked by an iPhone's TruthDepth camera and rendered locally using an articulated display. Our implementation can track the wide range of head movements that occur when one switches attention between a physical artifact, a laptop, and their collaborator. To support possible discrepency between local and remote layout, head movements are remapped as robotic movements to correspond to these key elements in the local user's space. We report on the results of an evaluation characterizing the system's remapping accuracy and its ability to support peripheral awareness of a remote participant's locus of attention.2022MSMose Sakashita et al.Remote and Hybrid CollaborationsCSCW
WovenProbe: Probing Possibilities for Weaving Fully-Integrated On-Skin Systems Deployable in the FieldOn-skin interfaces demonstrate great potential given their direct skin contact; however, conducting field studies of these devices outside of laboratories and in real settings remains a challenge. We conduct a research-through-design investigation using an extended woven practice for fabricating fully-integrated and untethered multi-sensor on-skin systems that are resilient, versatile, and capable of field deployment. We designed, implemented, and deployed a woven on-skin index-finger and thumb-based inertial measurement unit (IMU) sensing system for multi-hour use as a technology probe to understand the social, technical, and design facets towards moving integrated on-skin systems into a wearer’s daily life. Further, we integrate a woven NFC coil into the IMU on-skin system, which is wirelessly powered by a smartwatch substitute, signifying the potential of our woven approach for developing wirelessly powered on-skin systems for potential longer-term continuous wear. Our investigation and the lessons learned shed light on the opportunities for designing on-skin systems for everyday wear.2021KHKunpeng Huang et al.On-Skin Display & On-Skin InputDIS
C-Face: Continuously reconstructing facial expressions by deep learning contours of the face with ear-mounted miniature camerasC-Face (Contour-Face) is an ear-mounted wearable sensing technology that uses two miniature cameras to continuously reconstruct facial expressions by deep learning contours of the face. When facial muscles move, the contours of the face change from the point of view of the ear-mounted cameras. These subtle changes are fed into a deep learning model which continuously outputs 42 facial feature points representing the shapes and positions of the mouth, eyes and eyebrows. To evaluate C-Face, we embedded our technology into headphones and earphones. We conducted a user study with nine participants. In this study, we compared the output of our system to the feature points outputted by a state of the art computer vision library (Dlib1) from a font facing camera. We found that the mean error of all 42 feature points was 0.77 mm for earphones and 0.74 mm for headphones. The mean error for 20 major feature points capturing the most active areas of the face was 1.43 mm for earphones and 1.39 mm for headphones. The ability to continuously reconstruct facial expressions introduces new opportunities in a variety of applications. As a demonstration, we implemented and evaluated C-Face for two applications: facial expression detection (outputting emojis) and silent speech recognition. We further discuss the opportunities and challenges of deploying C-Face in real-world applications.2020TCTuochao Chen et al.Haptic WearablesHuman Pose & Activity RecognitionUIST
Ondulé: Designing and Controlling 3D Printable SpringsWe present Ondulé, a novel computational design tool to add elastic deformation behaviors to static 3D models using a combination of 3D-printed springs and mechanical joints. Springs are unique because they can exert expressive deformation behaviors and store mechanical energy. Informed by spring theory and our empirical mechanical experiments, we introduce spring and joint-based design techniques that support a range of parameterizable deformation behaviors, including compress, extend, twist, bend, and various combinations. To enable users to design and add these 3D-printable deformations to their models, we introduce a custom spring design tool for Rhino. Here, users can convert selected geometries into springs, customize spring stiffness, and parameterize their design to obtain a desired deformation behavior. To demonstrate the feasibility of our approach and the breadth of new 3D-printable designs that it enables, we showcase a set of example applications from launching rocket toys to tangible storytelling props. We conclude with a discussion of key challenges and open research questions.2019LHLiang He et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingUIST