LuxAct: Enhance Everyday Objects for Visual Sensing with Interaction-Powered Illumination

DOI

Imbuing sensing and interactivity into everyday objects has long been sought after within the HCI community to facilitate richer and more immersive user experiences. However, conventional methods rely on costly hardware, such as embedded sensor tags, or passive visual markers that lack digital capabilities to sense user context. We present LuxAct, an interaction-powered visual communication system that enables everyday objects to encode their information and user interaction data into sequences of RGB color light. These sequences are decoded by Point of View (POV) cameras on AR headsets or smart glasses to derive meaningful information from interactions. LuxAct are self-powered and ultra-low-cost, leveraging striking and plucking on piezoelectric generators to harvest energy from user interactions. Through strategic pattern design, our system transforms visual channels into carriers of both object identification and sensory data, supporting applications with rich sensing needs. We demonstrated a wide range of use cases, including interactive controls, sensate storage, smart water hose, medicine reminder, fingertip probes and beyond, offering a practical alternative for digitalizing passive objects to enable ubiquitous sensing in AR-enhanced environments.