Computational Design of Active Kinesthetic GarmentsGarments with the ability to provide kinesthetic force-feedback on-demand can augment human capabilities in a non-obtrusive way, enabling numerous applications in VR haptics, motion assistance, and robotic control. However, designing such garments is a complex, and often manual task, particularly when the goal is to resist multiple motions with a single design. In this work, we propose a computational pipeline for designing connecting structures between active components---one of the central challenges in this context. We focus on electrostatic (ES) clutches that are compliant in their passive state while strongly resisting elongation when activated. Our method automatically computes optimized connecting structures that efficiently resist a range of pre-defined body motions on demand. We propose a novel dual-objective optimization approach to simultaneously maximize the resistance to motion when clutches are active, while minimizing resistance when inactive. We demonstrate our method on a set of problems involving different body sites and a range of motions. We further fabricate and evaluate a subset of our automatically created designs against manually created baselines using mechanical testing and in a VR pointing study.2022VVVelko Vechev et al.Force Feedback & Pseudo-Haptic WeightHaptic WearablesFull-Body Interaction & Embodied InputUIST
Optimization-based User Support for Cinematographic Quadrotor Camera Target FramingTo create aesthetically pleasing aerial footage, the correct framing of camera targets is crucial. However, current quadrotor camera tools do not consider the 3D extent of actual camera targets in their optimization schemes and simply interpolate between keyframes when generating a trajectory. This can yield videos with aesthetically unpleasing target framing. In this paper, we propose an optimization formulation that optimizes the quadrotor camera pose such that targets are positioned at desirable screen locations according to videographic compositional rules and entirely visible throughout a shot. Camera targets are identified using a semi-automatic pipeline which leverages a deep-learning-based visual saliency model. A large-scale perceptual study (N≈500) shows that our method enables users to produce shots with a target framing that is closer to what they intended to create and more or as aesthetically pleasing than with the previous state of the art.2021CGChristoph Gebhardt et al.ETH ZurichDrone Interaction & ControlCHI
Omni: Volumetric Sensing and Actuation of Passive Magnetic Tools for Dynamic Haptic FeedbackWe present Omni, a self-contained 3D haptic feedback system that is capable of sensing and actuating an untethered, passive tool containing only a small embedded permanent magnet. Omni enriches AR, VR and desktop applications by providing an active haptic experience using a simple apparatus centered around an electromagnetic base. The spatial haptic capabilities of Omni are enabled by a novel gradient-based method to reconstruct the 3D position of the permanent magnet in midair using the measurements from eight off-the-shelf hall sensors that are integrated into the base. Omni’s 3 DoF spherical electromagnet simultaneously exerts dynamic and precise radial and tangential forces in a volumetric space around the device. Since our system is fully integrated, contains no moving parts and requires no external tracking, it is easy and affordable to fabricate. We describe Omni’s hardware implementation, our 3D reconstruction algorithm, and evaluate the tracking and actuation performance in depth. Finally, we demonstrate its capabilities via a set of interactive usage scenarios.2020TLThomas Langerak et al.Force Feedback & Pseudo-Haptic WeightShape-Changing Interfaces & Soft Robotic MaterialsFull-Body Interaction & Embodied InputUIST
Optimal Control for Electromagnetic Haptic Guidance SystemsWe introduce an optimal control method for electromagnetic haptic guidance systems. Our real-time approach assists users in pen-based tasks such as drawing, sketching or designing. The key to our control method is that it guides users, yet does not take away agency. Existing approaches force the stylus to a continuously advancing setpoint on a target trajectory, leading to undesirable behavior such as loss of haptic guidance or unintended snapping. Our control approach, in contrast, gently pulls users towards the target trajectory, allowing them to always easily override the system to adapt their input spontaneously and draw at their own speed. To achieve this fexible guidance, our optimization iteratively predicts the motion of an input device such as a pen, and adjusts the position and strength of an underlying dynamic electromagnetic actuator accordingly. To enable real-time computation, we additionally introduce a novel and fast approximate model of an electromagnet. We demonstrate the applicability of our approach by implementing it on a prototypical hardware platform based on an electromagnet moving on a bi-axial linear stage, as well as a set of applications. Experimental results show that our approach is more accurate and preferred by users compared to open-loop and time-dependent closed-loop approaches.2020TLThomas Langerak et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Force Feedback & Pseudo-Haptic WeightUIST
Context-Aware Online Adaptation of Mixed Reality InterfacesWe present an optimization-based approach for Mixed Reality (MR) systems to automatically control when and where applications are shown, and how much information they display. Currently, content creators design applications, and users then manually adjust which applications are visible and how much information they show. This choice has to be adjusted every time users switch context, i.e. whenever they switch their task or environment. Since context switches happen many times a day, we believe that MR interfaces require automation to alleviate this problem. We propose a real-time approach to automate this process based on users' current cognitive load, and knowledge about their task and environment. Our system adapts which applications are displayed, how much information they show, and where they are placed. We formulate this problem as a mix of rule-based decision making and combinatorial optimization which can be solved efficiently in real-time. We present a set of proof-of-concept applications showing that our approach is applicable in a wide range of scenarios. Finally, we present an evaluation with a dual task paradigm. Our approach resulted in similar task performance as a traditional UI, and decreased secondary tasks interactions by 36%.2019DLDavid Lindlbauer et al.AR Navigation & Context AwarenessContext-Aware ComputingUIST
Learning Cooperative Personalized Policies from Gaze DataAn ideal Mixed Reality (MR) system would only present virtual information (e.g., a label) when it is useful to the person. However, figuring out when a label is useful is challenging; it depends on a variety of factors, including the current task, previous knowledge, context, etc. In this paper, we propose a Reinforcement Learning (RL) method to learn when to show or hide an object’s label given eye movement data. We demonstrate the capabilities of this approach by showing that an intelligent agent can learn cooperative policies that better support users in a visual search task than design heuristics. Furthermore, we show the applicability of our approach in realistic environments and use cases (e.g., grocery shopping). By posing MR object labeling as an RL control problem we can learn policies implicitly by observing users’ behavior without requiring experience sampling or any other form of supervision.2019CGChristoph Gebhardt et al.Eye Tracking & Gaze InteractionMixed Reality WorkspacesAI-Assisted Decision-Making & AutomationUIST
AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-TimeDeveloping cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.2018SPSeonwook Park et al.ETH ZurichMixed Reality WorkspacesCreative Collaboration & Feedback SystemsCHI
Computational Interaction: Theory and PracticeThis course introduces computational methods in human--computer interaction. Computational interaction methods use computational thinking -- abstraction, automation, and analysis -- to explain and enhance interaction. This course introduces optimization and probabilistic inference as principled methods. Lectures center on hands-on Python programming, interleaving theory and practical examples.2018JWJulie R. Williamson et al.University of GlasgowComputational Methods in HCICHI
Computational Interaction: Theory and PracticeThis course introduces computational methods in human--computer interaction. Computational interaction methods use computational thinking -- abstraction, automation, and analysis -- to explain and enhance interaction. This course introduces optimization and probabilistic inference as principled methods. Lectures center on hands-on Python programming, interleaving theory and practical examples.2018JWJulie R. Williamson et al.University of GlasgowComputational Methods in HCICHI
Computational Interaction: Theory and PracticeThis course introduces computational methods in human--computer interaction. Computational interaction methods use computational thinking -- abstraction, automation, and analysis -- to explain and enhance interaction. This course introduces optimization and probabilistic inference as principled methods. Lectures center on hands-on Python programming, interleaving theory and practical examples.2018JWJulie R. Williamson et al.University of GlasgowProgramming Education & Computational ThinkingComputational Methods in HCICHI
DextrES: Wearable Haptic Feedback for Grasping in VR via a Thin Form-Factor Electrostatic BrakeWe introduce DextrES, a flexible and wearable haptic glove which integrates both kinesthetic and cutaneous feedback in a thin and light form factor (weight is less than 8g). Our approach is based on an electrostatic clutch generating up to 20 N of holding force on each finger by modulating the electrostatic attraction between flexible elastic metal strips to generate an electrically-controlled friction force. We harness the resulting braking force to rapidly render on-demand kinesthetic feedback. The electrostatic brake is mounted onto the the index finger and thumb via modular 3D printed articulated guides which allow the metal strips to glide smoothly. Cutaneous feedback is provided via piezo actuators at the fingertips. We demonstrate that our approach can provide rich haptic feedback under dexterous articulation of the user's hands and provides effective haptic feedback across a variety of different grasps. A controlled experiment indicates that DextrES improves the grasping precision for different types of virtual objects. Finally, we report on results of a psycho-physical study which identifies discrimination thresholds for different levels of holding force.2018RHRonan Hinchet et al.Vibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightHaptic WearablesUIST
DeepWriting: Making Digital Ink Editable via Deep Generative ModelingDigital ink promises to combine the flexibility and aesthetics of handwriting and the ability to process, search and edit digital text. Character recognition converts handwritten text into a digital representation, albeit at the cost of losing personalized appearance due to the technical difficulties of separating the interwoven components of content and style. In this paper, we propose a novel generative neural network architecture that is capable of disentangling style from content and thus making digital ink editable. Our model can synthesize arbitrary text, while giving users control over the visual appearance (style). For example, allowing for style transfer without changing the content, editing of digital ink at the word level and other application scenarios such as spell-checking and correction of handwritten text. We furthermore contribute a new dataset of handwritten text with fine-grained annotations at the character level and report results from an initial user evaluation.2018EAEmre Aksan et al.ETH ZurichGenerative AI (Text, Image, Music, Video)AI-Assisted Creative WritingCHI