IMUCoCo: Enabling Flexible On-Body IMU Placement for Human Pose Estimation and Activity RecognitionIMUs are regularly used to sense human motion, recognize activities, and estimate full-body pose. Users are typically required to place sensors in predefined locations that are often dictated by common wearable form factors and the machine learning model's training process. Consequently, despite the increasing number of everyday devices equipped with IMUs, the limited adaptability has seriously constrained the user experience to only using a few well-explored device placements (e.g., wrist and ears). In this paper, we rethink IMU-based motion sensing by acknowledging that signals can be captured from any point on the human body. We introduce IMU over Continuous Coordinates (IMUCoCo), a novel framework that maps signals from a variable number of IMUs placed on the body surface into a unified feature space based on their spatial coordinates. These features can be plugged into downstream models for pose estimation and activity recognition. Our evaluations demonstrate that IMUCoCo supports accurate pose estimation in a wide range of typical and atypical sensor placements. Overall, IMUCoCo supports significantly more flexible use of IMUs for motion sensing than the state-of-the-art, allowing users to place their sensors-laden devices according to their needs and preferences. The framework also supports the ability to change device locations depending on the context and suggests placement depending on the use case.2025HZHaozhe Zhou et al.Human Pose & Activity RecognitionUIST
Matcha: An IDE Plugin for Creating Accurate Privacy Nutrition LabelsLi 等人开发 Matcha IDE 插件，自动分析应用代码生成精确的隐私营养标签，帮助用户理解应用数据收集行为。2024TLTianshi Li et al.Privacy by Design & User ControlUbiComp
Kirigami: Lightweight Speech Filtering for Privacy-preserving Activity Recognition using AudioBoovaraghavan 等人设计 Kirigami 轻量级语音过滤框架，通过过滤敏感语音特征在保护用户隐私前提下实现音频活动识别。2024SBSudershan Boovaraghavan et al.Privacy by Design & User ControlUbiComp
ClassID: Enabling Student Behavior Attribution from Ambient Classroom Sensing SystemsPatidar 等人开发ClassID环境课堂感知系统，通过多模态传感器和机器学习实现学生行为自动归因，帮助教师实时了解课堂动态与学生参与度。2024PPPrasoon Patidar et al.Intelligent Tutoring Systems & Learning AnalyticsCollaborative Learning & Peer TeachingUser Research Methods (Interviews, Surveys, Observation)UbiComp
Bring Privacy To The Table: Interactive Negotiation for Privacy Settings of Shared Sensing DevicesTo address privacy concerns with the Internet of Things (IoT) devices, researchers have proposed enhancements in data collection transparency and user control. However, managing privacy preferences for shared devices with multiple stakeholders remains challenging. We introduced ThingPoll, a system that helps users negotiate privacy configurations for IoT devices in shared settings. We designed ThingPoll by observing twelve participants verbally negotiating privacy preferences, from which we identified potentially successful and inefficient negotiation patterns. ThingPoll bootstraps a preference model from a custom crowdsourced privacy preferences dataset. During negotiations, ThingPoll strategically scaffolds the process by eliciting users’ privacy preferences, providing helpful contexts, and suggesting feasible configuration options. We evaluated ThingPoll with 30 participants negotiating the privacy settings of 4 devices. Using ThingPoll, participants reached an agreement in 97.5% of scenarios within an average of 3.27 minutes. Participants reported high overall satisfaction of 83.3% with ThingPoll as compared to baseline approaches.2024HZHaozhe Zhou et al.Carnegie Mellon UniversityPrivacy by Design & User ControlIoT Device PrivacyCHI
Is a Trustmark and QR Code Enough? The Effect of IoT Security and Privacy Label Information Complexity on Consumer Comprehension and BehaviorThe U.S. Government is developing a package label to help consumers access reliable security and privacy information about Internet of Things (IoT) devices when making purchase decisions. The label will include the U.S. Cyber Trust Mark, a QR code to scan for more details, and potentially additional information. To examine how label information complexity and educational interventions affect comprehension of security and privacy attributes and label QR code use, we conducted an online survey with 518 IoT purchasers. We examined participants' comprehension and preferences for three labels of varying complexities, with and without an educational intervention. Participants favored and correctly utilized the two higher-complexity labels, showing a special interest in the privacy-relevant content. Furthermore, while the educational intervention improved understanding of the QR code’s purpose, it had a modest effect on QR scanning behavior. We highlight clear design and policy directions for creating and deploying IoT security and privacy labels.2024CCClaire C Chen et al.Carnegie Mellon UniversityPrivacy Perception & Decision-MakingSmart Home Privacy & SecurityCHI
Mites: Design and Deployment of a General-Purpose Sensing Infrastructure for Buildings"There is increasing interest in deploying building-scale, general-purpose, and high-fidelity sensing to drive emerging smart building applications. However, the real-world deployment of such systems is challenging due to the lack of system and architectural support. Most existing sensing systems are purpose-built, consisting of hardware that senses a limited set of environmental facets, typically at low fidelity and for short-term deployment. Furthermore, prior systems with high-fidelity sensing and machine learning fail to scale effectively and have fewer primitives, if any, for privacy and security. For these reasons, IoT deployments in buildings are generally short-lived or done as a proof of concept. We present the design of Mites, a scalable end-to-end hardware-software system for supporting and managing distributed general-purpose sensors in buildings. Our design includes robust primitives for privacy and security, essential features for scalable data management, as well as machine learning to support diverse applications in buildings. We deployed our Mites system and 314 Mites devices in Tata Consultancy Services (TCS) Hall at Carnegie Mellon University (CMU), a fully occupied, five-story university building. We present a set of comprehensive evaluations of our system using a series of microbenchmarks and end-to-end evaluations to show how we achieved our stated design goals. We include five proof-of-concept applications to demonstrate the extensibility of the Mites system to support compelling IoT applications. Finally, we discuss the real-world challenges we faced and the lessons we learned over the five-year journey of our stack's iterative design, development, and deployment. https://dl.acm.org/doi/10.1145/3580865"2023SBSudershan Boovaraghavan et al.Context-Aware ComputingSmart Home Privacy & SecuritySmart Cities & Urban SensingUbiComp
TAO: Context Detection from Daily Activity Patterns Using Temporal Analysis and Ontology"Translating fine-grained activity detection (e.g., phone ring, talking interspersed with silence and walking) into semantically meaningful and richer contextual information (e.g., on a phone call for 20 minutes while exercising) is essential towards enabling a range of healthcare and human-computer interaction applications. Prior work has proposed building ontologies or temporal analysis of activity patterns with limited success in capturing complex real-world context patterns. We present TAO, a hybrid system that leverages OWL-based ontologies and temporal clustering approaches to detect high-level contexts from human activities. TAO can characterize sequential activities that happen one after the other and activities that are interleaved or occur in parallel to detect a richer set of contexts more accurately than prior work. We evaluate TAO on real-world activity datasets (Casas and Extrasensory) and show that our system achieves, on average, 87% and 80% accuracy for context detection, respectively. We deploy and evaluate TAO in a real-world setting with eight participants using our system for three hours each, demonstrating TAO's ability to capture semantically meaningful contexts in the real world. Finally, to showcase the usefulness of contexts, we prototype wellness applications that assess productivity and stress and show that the wellness metrics calculated using contexts provided by TAO are much closer to the ground truth (on average within 1.1%), as compared to the baseline approach (on average within 30%)." https://doi.org/10.1145/36108962023SBSudershan Boovaraghavan et al.Human Pose & Activity RecognitionContext-Aware ComputingUbiComp
VAX: Using Existing Video and Audio-based Activity Recognition Models to Bootstrap Privacy-Sensitive Sensors"The use of audio and video modalities for Human Activity Recognition (HAR) is common, given the richness of the data and the availability of pre-trained ML models using a large corpus of labeled training data. However, audio and video sensors also lead to significant consumer privacy concerns. Researchers have thus explored alternate modalities that are less privacy-invasive such as mmWave doppler radars, IMUs, motion sensors. However, the key limitation of these approaches is that most of them do not readily generalize across environments and require significant in-situ training data. Recent work has proposed cross-modality transfer learning approaches to alleviate the lack of trained labeled data with some success. In this paper, we generalize this concept to create a novel system called VAX (Video/Audio to 'X'), where training labels acquired from existing Video/Audio ML models are used to train ML models for a wide range of 'X' privacy-sensitive sensors. Notably, in VAX, once the ML models for the privacy-sensitive sensors are trained, with little to no user involvement, the Audio/Video sensors can be removed altogether to protect the user's privacy better. We built and deployed VAX in ten participants' homes while they performed 17 common activities of daily living. Our evaluation results show that after training, VAX can use its onboard camera and microphone to detect approximately 15 out of 17 activities with an average accuracy of 90%. For these activities that can be detected using a camera and a microphone, VAX trains a per-home model for the privacy-preserving sensors. These models (average accuracy = 84%) require no in-situ user input. In addition, when VAX is augmented with just one labeled instance for the activities not detected by the VAX A/V pipeline (~2 out of 17), it can detect all 17 activities with an average accuracy of 84%. Our results show that VAX is significantly better than a baseline supervised-learning approach of using one labeled instance per activity in each home (average accuracy of 79%) since VAX reduces the user burden of providing activity labels by 8x (~2 labels vs. 17 labels)." https://doi.org/10.1145/36109072023PPPrasoon Patidar et al.Human Pose & Activity RecognitionBiosensors & Physiological MonitoringContext-Aware ComputingUbiComp
"An Instructor is [already] able to keep track of 30 students": Students’ Perceptions of Smart Classrooms for Improving Teaching & Their Emergent Understandings of Teaching and LearningMulti-modal classroom sensing systems can collect complex behaviors in the classroom at a scale and precision far greater than human observers to capture learning insights and provide personalized teaching feedback. As students are critical stakeholders in the adoption of smart classrooms for the improvement of teaching, open questions remain in understanding student perspectives on the use of their data to provide insights to instructors. We conducted a Speed Dating with storyboards study to explore student values and boundaries regarding the acceptance of classroom sensing systems in STEM college courses. We found that students have several emergent beliefs about teaching and learning that influence their views towards smart classroom technologies. Students also held contextual views on the boundaries of data use depending on the outcome. Our findings have implications for the design and communication of classroom sensing systems that reconcile student and instructor beliefs around teaching and learning.2023TNTricia J. Ngoon et al.Intelligent Tutoring Systems & Learning AnalyticsSTEM Education & Science CommunicationDIS
Exploring the Needs of Users for Supporting Privacy-protective Behavior in Smart HomesIn this paper, we studied people’s smart home privacy-protective behaviors (SH-PPBs), to gain a better understanding of their privacy management do’s and don’ts in this context. We first surveyed 159 participants and elicited 33 unique SH-PPB practices, revealing that users heavily rely on ad hoc approaches at the physical layer (e.g., physical blocking, manual powering off). We also characterized the types of privacy concerns users wanted to address through SH-PPBs, the reasons preventing users from doing SH-PPBs, and privacy features they wished they had to support SH-PPBs. We then storyboarded 11 privacy protection concepts to explore opportunities to better support users’ needs, and asked another 227 participants to criticize and rank these design concepts. Among the 11 concepts, Privacy Diagnostics, which is similar to security diagnostics in anti-virus software, was far preferred over the rest. We also witnessed rich evidence of four important factors in designing SH-PPB tools, as users prefer (1) simple, (2) proactive, (3) preventative solutions that can (4) offer more control.2022HJHaojian Jin et al.CMUAlgorithmic Transparency & AuditabilityPrivacy by Design & User ControlSmart Home Privacy & SecurityCHI
Understanding Challenges for Developers to Create Accurate Privacy Nutrition LabelsApple announced the introduction of app privacy details to their App Store in December 2020, marking the first ever real-world, large-scale deployment of the privacy nutrition label concept, which had been introduced by researchers over a decade earlier. The Apple labels are created by app developers, who self-report their app's data practices. In this paper, we present the first study examining the usability and understandability of Apple's privacy nutrition label creation process from the developer's perspective. By observing and interviewing 12 iOS app developers about how they created the privacy label for a real-world app that they developed, we identified common challenges for correctly and efficiently creating privacy labels. We discuss design implications both for improving Apple's privacy label design and for future deployment of other standardized privacy notices.2022TLTianshi Li et al.Carnegie Mellon University, Carnegie Mellon UniversityPrivacy by Design & User ControlPrivacy Perception & Decision-MakingCHI
Classroom Digital Twins with Instrumentation-Free Gaze TrackingClassroom sensing is an important and active area of research with great potential to improve instruction. Complementing professional observers - the current best practice - automated pedagogical professional development systems can attend every class and capture fine-grained details of all occupants. One particularly valuable facet to capture is class gaze behavior. For students, certain gaze patterns have been shown to correlate with interest in the material, while for instructors, student-centered gaze patterns have been shown to increase approachability and immediacy. Unfortunately, prior classroom gaze-sensing systems have limited accuracy and often require specialized external or worn sensors. In this work, we developed a new computer-vision-driven system that powers a 3D “digital twin” of the classroom and enables whole-class, 6DOF head gaze vector estimation without instrumenting any of the occupants. We describe our open source implementation, and results from both controlled studies and real-world classroom deployments.2021KAKaran Ahuja et al.Carnegie Mellon UniversityEye Tracking & Gaze InteractionMixed Reality WorkspacesCHI
Exploring How Privacy and Security Factor into IoT Device Purchase BehaviorDespite growing concerns about security and privacy of Internet of Things (IoT) devices, consumers generally do not have access to security and privacy information when purchasing these devices. We interviewed 24 participants about IoT devices they purchased. While most had not considered privacy and security prior to purchase, they reported becoming concerned later due to media reports, opinions shared by friends, or observing unexpected device behavior. Those who sought privacy and security information before purchase, reported that it was difficult or impossible to find. We asked interviewees to rank factors they would consider when purchasing IoT devices; after features and price, privacy and security were ranked among the most important. Finally, we showed interviewees our prototype privacy and security label. Almost all found it to be accessible and useful, encouraging them to incorporate privacy and security in their IoT purchase decisions.2019PEPardis Emami-Naeini et al.Carnegie Mellon UniversityPrivacy by Design & User ControlPrivacy Perception & Decision-MakingCHI