Opisthenar: Hand Poses and Finger Tapping Recognition by Observing Back of Hand Using Embedded Wrist Camera

We introduce a vision-based technique to recognize hand poses and gestures by simply observing changes on the back of the hand. Our approach employs a camera on the wrist, which we envisage can be included in a wrist-worn device such as a smartwatch, fitness tracker or wristband. However, in this configuration the fingers are occluded from the view of the camera. The oblique angle and placement of the camera make typical vision-based techniques difficult to adopt. Our alternative approach observes small changes and movements in the shape, tendons, skin and bone on the back of the hand. We uses a deep neural network to train and recognize both static hand poses and dynamic gestures. While this is a challenging configuration for sensing, we tested the recognition with a real-time user test and can achieve a high recognition rate of 89.4% (static) and 67.5% (dynamic). Our results further demonstrate that our approach can generalize across sessions and to new users. Namely, users can remove and replace the wrist-worn device while new users can employ a previously trained system, to a certain extent. This form of sensing affords a range of new interaction capabilities from one-handed to subtle inputs or eyes-free to orientation invariant interactions.