How Do We Interact With the World?

Understanding vision, cognition and action requires the integrated investigation of behavioral, neural and computational processes

How do humans interact with the world? Humans are immensely complex systems who are able to receive and transform a continual barrage of sensory information into seemingly effortless, willful motor actions. This impressive job, allocated to the nervous system, remains a major unresolved scientific puzzle. A major goal of Brown’s Brain Science Program is to research the behavioral, neural, and computational processes that allow humans to move about the world. Many of them focus upon visual-to-motor interactions in the formation of accurate and purposeful actions. Voluntary actions commonly involve the use of environmental visual cues to plan an action, to achieve a goal accurately, to perform gestures, or to navigate in the world. Visual information can be used in flexible ways and can enhance (or degrade) our actions—imagine how a baseball batter reads all of the signs of the pitcher and the motions of an oncoming ball to precisely guide the bat with, in some cases, unbelievably precise control. Understanding how this kind of processing is accomplished, at the scientific level, involves experimental manipulation of input and output requirements to explore the brain’s mechanisms used to generate these sensory to motor transformations.

Using functional Magnetic Resonance Imaging (fMRI), students in Professor Jerome Sanes’ laboratory at Brown have discovered synergies between eye position and brain activity related to moving the hand. Visual attention may drive the effects of eye position on finger movements since our direction of gaze and our visual attention are often linked. To address this issue, subjects performed movements and visual tasks that were unrelated. Despite this behavioral independence, novel brain representations emerged during simultaneous performance of the two tasks, particularly in the parietal lobe and orbital frontal cortex, where humans process space. This result is illustrated in the fMRI images shown here. In the left panel, the regions depicted in red indicate higher brain activity for finger movement compared to visual attention, whereas the green regions represent more activation for visual attention. The right panel illustrates concurrent effects of visual attention and movement. The parietal area in green indicates where attention-related activation was greater during movement. In contrast, the two areas in red show where attention-related activation was diminished during movement.

To investigate how we interact with the world, Professor William Warren and his students are applying cutting-edge virtual reality technology to the scientific study of perception and action. Research in Brown’s Virtual Environment Navigation Lab (VENLab) investigates how observers visually control their locomotion and navigation in complex environments. Subjects wearing a head-mounted display can walk freely in 40 x 40 ft. room while immersed in a highly realistic computer-generated virtual environment (Brown’s VENLab is the largest fully immersive virtual reality lab in the world and the first one located at a U.S. university). One line of experiments studies the visual strategies people use to steer toward stationary and moving targets and to avoid stationary and moving obstacles. The results reveal that we use patterns of visual motion, called optic flow, to guide our walking. A dynamical model of steering and obstacle avoidance demonstrates that the routes people take through complex scenes can emerge from their interactions with the environment, without explicit planning.

Navigation over longer distances, however, depends on some spatial knowledge of the layout of the environment. Research in the VENLab investigates the structure of our "cognitive maps" by manipulating the properties of a virtual environment during active navigation. After subjects learn the layout of a "secret garden", the virtual environment is stretched or rotated before they find their way to a known location. Results show that people rely more on the ordinal structure ("take the second right") than the metric structure ("go 20 feet and turn right") of the environment for navigation. Such spatial knowledge interacts with other navigation strategies such as path integration and use of landmarks. When, after learning a route, landmarks along it are displaced, the subjects’ paths are shifted about halfway in the same direction. As these biological strategies for locomotion and navigation are studied in people, they are also implemented in mobile robots, which simultaneously provide a test bed for and suggest new hypotheses about human behavior. These findings suggest ways that the brain codes for such processing. Work in Dr. Mayank Mehta’s lab investigates the neural codes for navigation at the cellular level by studying how neurons in the hippocampus of animals capture images of ‘place’ as the animals navigate through the world.

Computer scientists at Brown are developing ways to interact and analyze complex models of the brain in immersive virtual reality (IVR) environment created in Brown’s CAVE facility. This virtual reality facility is used to facilitate our understanding of the three-dimensional structure of connections within the brain. The visualizations can be used in far-ranging ways—to plan brain surgery; to minimize complications when removing a tumor, or; to track the progression of neuro-degenerative disorders in the brain. By testing general user response and the ability to visually isolate important visualization characteristics, scientists are progressively creating better ways to interact productively with these virtual environments. Such processes are increasingly necessary to help analyze emerging larger and more complex datasets that can be produced from studies of the healthy and diseased nervous system.

Posted 11/03









				How Do We Interact With the World? Understanding vision, cognition and action requires the integrated investigation of behavioral, neural and computational processes How do humans interact with the world? Humans are immensely complex systems who are able to receive and transform a continual barrage of sensory information into seemingly effortless, willful motor actions. This impressive job, allocated to the nervous system, remains a major unresolved scientific puzzle. A major goal of Brown’s Brain Science Program is to research the behavioral, neural, and computational processes that allow humans to move about the world. Many of them focus upon visual-to-motor interactions in the formation of accurate and purposeful actions. Voluntary actions commonly involve the use of environmental visual cues to plan an action, to achieve a goal accurately, to perform gestures, or to navigate in the world. Visual information can be used in flexible ways and can enhance (or degrade) our actions—imagine how a baseball batter reads all of the signs of the pitcher and the motions of an oncoming ball to precisely guide the bat with, in some cases, unbelievably precise control. Understanding how this kind of processing is accomplished, at the scientific level, involves experimental manipulation of input and output requirements to explore the brain’s mechanisms used to generate these sensory to motor transformations. Using functional Magnetic Resonance Imaging (fMRI), students in Professor Jerome Sanes’ laboratory at Brown have discovered synergies between eye position and brain activity related to moving the hand. Visual attention may drive the effects of eye position on finger movements since our direction of gaze and our visual attention are often linked. To address this issue, subjects performed movements and visual tasks that were unrelated. Despite this behavioral independence, novel brain representations emerged during simultaneous performance of the two tasks, particularly in the parietal lobe and orbital frontal cortex, where humans process space. This result is illustrated in the fMRI images shown here. In the left panel, the regions depicted in red indicate higher brain activity for finger movement compared to visual attention, whereas the green regions represent more activation for visual attention. The right panel illustrates concurrent effects of visual attention and movement. The parietal area in green indicates where attention-related activation was greater during movement. In contrast, the two areas in red show where attention-related activation was diminished during movement. To investigate how we interact with the world, Professor William Warren and his students are applying cutting-edge virtual reality technology to the scientific study of perception and action. Research in Brown’s Virtual Environment Navigation Lab (VENLab) investigates how observers visually control their locomotion and navigation in complex environments. Subjects wearing a head-mounted display can walk freely in 40 x 40 ft. room while immersed in a highly realistic computer-generated virtual environment (Brown’s VENLab is the largest fully immersive virtual reality lab in the world and the first one located at a U.S. university). One line of experiments studies the visual strategies people use to steer toward stationary and moving targets and to avoid stationary and moving obstacles. The results reveal that we use patterns of visual motion, called optic flow, to guide our walking. A dynamical model of steering and obstacle avoidance demonstrates that the routes people take through complex scenes can emerge from their interactions with the environment, without explicit planning. Navigation over longer distances, however, depends on some spatial knowledge of the layout of the environment. Research in the VENLab investigates the structure of our "cognitive maps" by manipulating the properties of a virtual environment during active navigation. After subjects learn the layout of a "secret garden", the virtual environment is stretched or rotated before they find their way to a known location. Results show that people rely more on the ordinal structure ("take the second right") than the metric structure ("go 20 feet and turn right") of the environment for navigation. Such spatial knowledge interacts with other navigation strategies such as path integration and use of landmarks. When, after learning a route, landmarks along it are displaced, the subjects’ paths are shifted about halfway in the same direction. As these biological strategies for locomotion and navigation are studied in people, they are also implemented in mobile robots, which simultaneously provide a test bed for and suggest new hypotheses about human behavior. These findings suggest ways that the brain codes for such processing. Work in Dr. Mayank Mehta’s lab investigates the neural codes for navigation at the cellular level by studying how neurons in the hippocampus of animals capture images of ‘place’ as the animals navigate through the world. Computer scientists at Brown are developing ways to interact and analyze complex models of the brain in immersive virtual reality (IVR) environment created in Brown’s CAVE facility. This virtual reality facility is used to facilitate our understanding of the three-dimensional structure of connections within the brain. The visualizations can be used in far-ranging ways—to plan brain surgery; to minimize complications when removing a tumor, or; to track the progression of neuro-degenerative disorders in the brain. By testing general user response and the ability to visually isolate important visualization characteristics, scientists are progressively creating better ways to interact productively with these virtual environments. Such processes are increasingly necessary to help analyze emerging larger and more complex datasets that can be produced from studies of the healthy and diseased nervous system. Posted 11/03