Visual Computations for Perception and Action
David Knill
- "We do not see optical images in an optical space, but we perceive the bodies round about us in their many and sensuous qualities."
- —Ernst Mach, 1897
Mach's quote, taken from a short monograph on sensation and perception, states very beautifully the basic phenomenon of vision - that our perceptions are comprised of descriptions of the world, not of the energy patterns directly available to our senses. A central puzzle in vision is how the brain infers properties of the world from the sensory data. Research in my lab focuses on two aspects of the vision problem: how the human visual system perceptually infers physical and geometric properties of the world from image data and how the brain uses visual information about the world to control motor behavior, such as reaching for and grasping objects. Answering these questions requires understanding the relationships between visual processing, the structure of visual information and the information demands of the tasks subserved by vision. Research in my lab, therefore, takes an integrative approach that combines mathematical analysis and psychophysical experimentation.
One central problem being studied in my lab is how the visual system perceives the 3-D shapes and layouts of surfaces in a scene. Our theoretical work in this area focuses on analyzing how regularities in the world help to structure the information provided by different cues and how multiple cues should (ideally) be integrated to generate percepts of 3-D surface geometry. The results of this "computational" analysis guide psychophysical investigations into how the visual system uses the information provided by different cues to infer 3-D scene structure. We have applied this approach to a number of specific problems, including the perception of 3-D object motion from the motion of cast shadows, the perception of surface shape from contours, the perception of surface shape from texture and the perception of shape from shading. We are currently studying how the visual system integrates the information from different cues to generate a unified percept of surface shape. This work attempts to relate theoretical analyses of ideal cue integration strategies to the strategies used by human observers as revealed by psychophysical studies.
A second general problem domain being investigated in my lab is how the human brain uses visual information to control motor behavior, particularly prehension movements (reaching for and grasping objects). The lab contains a 3-D display system for generating a virtual world with which subjects can interact, a robot arm for automatically placing real targets in an experiment Wanda real-time 3-D sensing system for recording subjects' finger, hand and arm movements during execution of a reaching task. By artificially perturbing parts of the virtual 3-D world during a prehension movement, we are able to study how the human brain uses visual information to control the movement. One problem currently being investigated is how the brain integrates different visual cues to the 3-D position, orientation and shape of objects to control reaching and grasping movements and whether the integration strategies for motor control differ from those discovered for perception. Another is how the brain combines information about the 3-D geometry of a target object with visual feedback from the moving hand to control prehension movements. This work allows us to close the loop between perception and action and allows us to study visual processing in the context of a natural task for which it has evolved.