The presentation Maestro: Direct manipulation through gesture alone

Abstract

Past research suggests a number of benefits to using hand-based interaction when interacting with electronic presentations. This paper introduces Maestro, a computer-vision based presentation system that uses hand gestures to allow fine-grained interaction with the contests of a projected slideshow. Maestro employs a single web camera and no other physical mediators. The contributions of this paper lie in robustly solving the gesture segmentation problem inherent in using only computer vision as input, and in a set of feedback mechanisms designed to scaffold use of the recognition system without interfering with the visual presentation of content. Specifically, Maestro employs bimanual cues, spatial and content based context, hand roles, and, in some case, dwell time to segment gestures. These strong segmentation cues are robust with respect to false positives and allows the actual gestures themselves to more directly match the action to be performed. They also result in a direct manipulation-style interface built on gestures alone. To assist with use, Maestro introduces a feedback comet, an augmented cursor that provides mnemonics for gestures and feedback to help govern gesture speeds to those conducive to gesture recognition. We present the design of the system and lessons learned from user evaluations.