A neural representation of depth from motion parallax in macaque visual cortex

Perception of depth is a fundamental challenge for the visual system, particularly for observers moving through their environment. The brain makes use of multiple visual cues to reconstruct the three-dimensional structure of a scene. One potent cue, motion parallax, frequently arises during translation of the observer because the images of objects at different distances move across the retina with different velocities. Human psychophysical studies have demonstrated that motion parallax can be a powerful depth cue, and motion parallax seems to be heavily exploited by animal species that lack highly developed binocular vision. However, little is known about the neural mechanisms that underlie this capacity. Here we show, by using a virtual-reality system to translate macaque monkeys (Macaca mulatta) while they viewed motion parallax displays that simulated objects at different depths, that many neurons in the middle temporal area (area MT) signal the sign of depth (near versus far) from motion parallax in the absence of other depth cues. To achieve this, neurons must combine visual motion with extra-retinal (non-visual) signals related to the animal's movement. Our findings suggest a new neural substrate for depth perception and demonstrate a robust interaction of visual and non-visual cues in area MT. Combined with previous studies that implicate area MT in depth perception based on binocular disparities, our results suggest that area MT contains a more general representation of three-dimensional space that makes use of multiple cues.     

Guarding the gateway to cortex with attention in visual thalamus

The massive visual input from the eye to the brain requires selective processing of some visual information at the expense of other information, a process referred to as visual attention. Increases in the responses of visual neurons with attention have been extensively studied along the visual processing streams in monkey cerebral cortex, from primary visual areas to parietal and frontal cortex. Here we show, by recording neurons in attending macaque monkeys (Macaca mulatta), that attention modulates visual signals before they even reach cortex by increasing responses of both magnocellular and parvocellular neurons in the first relay between retina and cortex, the lateral geniculate nucleus (LGN). At the same time, attention decreases neuronal responses in the adjacent thalamic reticular nucleus (TRN). Crick argued for such modulation of the LGN by observing that it is inhibited by the TRN, and suggested that "if the thalamus is the gateway to the cortex, the reticular complex might be described as the guardian of the gateway", a reciprocal relationship we now show to be more than just hypothesis. The reciprocal modulation in LGN and TRN appears only during the initial visual response, but the modulation of LGN reappears later in the response, suggesting separate early and late sources of attentional modulation in LGN.     

Perceptual consequences of centre-surround antagonism in visual motion processing

Centre-surround receptive field organization is a ubiquitous property in mammalian visual systems, presumably tailored for extracting image features that are differentially distributed over space. In visual motion, this is evident as antagonistic interactions between centre and surround regions of the receptive fields of many direction-selective neurons in visual cortex. In a series of psychophysical experiments we make the counterintuitive observation that increasing the size of a high-contrast moving pattern renders its direction of motion more difficult to perceive and reduces its effectiveness as an adaptation stimulus. We propose that this is a perceptual correlate of centre-surround antagonism, possibly within a population of neurons in the middle temporal visual area. The spatial antagonism of motion signals observed at high contrast gives way to spatial summation as contrast decreases. Evidently, integration of motion signals over space depends crucially on the visibility of those signals, thereby allowing the visual system to register motion information efficiently and adaptively.     

Segregation of global and local motion processing in primate middle temporal visual area.

The early stages of primate visual processing appear to be divided up into several component parts so that, for example, colour, form and motion are analysed by anatomically distinct streams. We have found that further subspecialization occurs within the motion processing stream. Neurons representing two different kinds of information about visual motion are segregated in columnar fashion within the middle temporal area of the owl monkey. These columns can be distinguished by labelling with 2-deoxyglucose in response to large-field random-dot patterns. Neurons in lightly labelled interbands have receptive fields with antagonistic surrounds: the response to a centrally placed moving stimulus is suppressed by motion in the surround. Neurons in more densely labelled bands have surrounds that reinforce the centre response so that they integrate motion cues over large areas of the visual field. Interband cells carry information about local motion contrast that may be used to detect motion boundaries or to indicate retinal slip during visual tracking. Band cells encode information about global motion that might be useful for orienting the animal in its environment.     

Influence of the thalamus on spatial visual processing in frontal cortex

Each of our movements activates our own sensory receptors, and therefore keeping track of self-movement is a necessary part of analysing sensory input. One way in which the brain keeps track of self-movement is by monitoring an internal copy, or corollary discharge, of motor commands. This concept could explain why we perceive a stable visual world despite our frequent quick, or saccadic, eye movements: corollary discharge about each saccade would permit the visual system to ignore saccade-induced visual changes. The critical missing link has been the connection between corollary discharge and visual processing. Here we show that such a link is formed by a corollary discharge from the thalamus that targets the frontal cortex. In the thalamus, neurons in the mediodorsal nucleus relay a corollary discharge of saccades from the midbrain superior colliculus to the cortical frontal eye field. In the frontal eye field, neurons use corollary discharge to shift their visual receptive fields spatially before saccades. We tested the hypothesis that these two components-a pathway for corollary discharge and neurons with shifting receptive fields-form a circuit in which the corollary discharge drives the shift. First we showed that the known spatial and temporal properties of the corollary discharge predict the dynamic changes in spatial visual processing of cortical neurons when saccades are made. Then we moved from this correlation to causation by isolating single cortical neurons and showing that their spatial visual processing is impaired when corollary discharge from the thalamus is interrupted. Thus the visual processing of frontal neurons is spatiotemporally matched with, and functionally dependent on, corollary discharge input from the thalamus. These experiments establish the first link between corollary discharge and visual processing, delineate a brain circuit that is well suited for mediating visual stability, and provide a framework for studying corollary discharge in other sensory systems.     

Hearing visual motion in depth

Auditory spatial perception is strongly affected by visual cues. For example, if auditory and visual stimuli are presented synchronously but from different positions, the auditory event is mislocated towards the locus of the visual stimulus-the ventriloquism effect. This 'visual capture' also occurs in motion perception in which a static auditory stimulus appears to move with the visual moving object. We investigated how the human perceptual system coordinates complementary inputs from auditory and visual senses. Here we show that an auditory aftereffect occurs from adaptation to visual motion in depth. After a few minutes of viewing a square moving in depth, a steady sound was perceived as changing loudness in the opposite direction. Adaptation to a combination of auditory and visual stimuli changing in a compatible direction increased the aftereffect and the effect of visual adaptation almost disappeared when the directions were opposite. On the other hand, listening to a sound changing in intensity did not affect the visual changing-size aftereffect. The results provide psychophysical evidence that, for processing of motion in depth, the auditory system responds to both auditory changing intensity and visual motion in depth.     

Receptive field dynamics in adult primary visual cortex.

The adult brain has a remarkable ability to adjust to changes in sensory input. Removal of afferent input to the somatosensory, auditory, motor or visual cortex results in a marked change of cortical topography. Changes in sensory activity can, over a period of months, alter receptive field size and cortical topography. Here we remove visual input by focal binocular retinal lesions and record from the same cortical sites before and within minutes after making the lesion and find immediate striking increases in receptive field size for cortical cells with receptive fields near the edge of the retinal scotoma. After a few months even the cortical areas that were initially silenced by the lesion recover visual activity, representing retinotopic loci surrounding the lesion. At the level of the lateral geniculate nucleus, which provides the visual input to the striate cortex, a large silent region remains. Furthermore, anatomical studies show that the spread of geniculocortical afferents is insufficient to account for the cortical recovery. The results indicate that the topographic reorganization within the cortex was largely due to synaptic changes intrinsic to the cortex, perhaps through the plexus of long-range horizontal connections.     

Grouping of image fragments in primary visual cortex.

In the visual world, objects are partially occluded by nearer objects, separating them into image fragments. However, the image fragments of the object can easily be grouped and organized together by the visual system. Psychophysical data and theoretical analysis indicate that such perceptual grouping might be mediated in the early stages of visual processing. Here I show that some orientation-selective cells in the primary visual cortex (V1) have response properties that can mediate the grouping of image fragments. These cells stopped responding to a stimulus bar when it was partly occluded by a small patch. The cells also did not respond when the patch had uncrossed disparity so that it appeared to be behind the bar. However, the cells began responding again when the patch had crossed disparity so that it appeared to be in front of the bar. These results indicate that cells as early as V1 have the computational power to make inferences about the nature of partially invisible forms seen behind occluding structures.     

Involvement of visual cortex in tactile discrimination of orientation.

The primary sense modalities (vision, touch and so on) are generally thought of as distinct. However, visual imagery is implicated in the normal tactile perception of some object properties, such as orientation, shape and size. Furthermore, certain tactile tasks, such as discrimination of grating orientation and object recognition, are associated with activity in areas of visual cortex. Here we show that disrupting function of the occipital cortex using focal transcranial magnetic stimulation (TMS) interferes with the tactile discrimination of grating orientation. The specificity of this effect is illustrated by its time course and spatial restriction over the scalp, and by the failure of occipital TMS to affect either detection of an electrical stimulus applied to the fingerpad or tactile discrimination of grating texture. In contrast, TMS over the somatosensory cortex blocked discrimination of grating texture as well as orientation. We also report that, during tactile discrimination of grating orientation, an evoked potential is recorded over posterior scalp regions with a latency corresponding to the peak of the TMS interference effect (about 180 ms). The findings indicate that visual cortex is closely involved in tactile discrimination of orientation. To our knowledge, this is the first demonstration that visual cortical processing is necessary for normal tactile perception.     

Segregation of pathways leading from area V2 to areas V4 and V5 of macaque monkey visual cortex

V5 and V4 are areas of macaque monkey prestriate visual cortex that are specialized for involvement in different aspects of visual perception, namely motion for V5 (refs 1-4) and colour vision, with other possible functions, for V4 (refs 2, 5-9). Thus, it is unlikely that they should be fed the same information for further processing, yet both receive a strong input from patches of the upper layers of V2 (refs 10, 11), the area immediately adjoining the primary visual cortex, V1. V2, however, seems to comprise functionally distinct subregions, which can be revealed by staining the tissue for the mitochondrial enzyme cytochrome oxidase. Here we report that V4 and V5 are connected with separate cytochrome oxidase-defined subregions of V2, suggesting that cortical pathways dealing with motion and colour perception are segregated in their passage through V2, and reinforcing evidence for functional specialization in the visual cortex.     

Linear processing of spatial cues in primary auditory cortex.

To determine the direction of a sound source in space, animals must process a variety of auditory spatial cues, including interaural level and time differences, as well as changes in the sound spectrum caused by the direction-dependent filtering of sound by the outer ear. Behavioural deficits observed when primary auditory cortex (A1) is damaged have led to the widespread view that A1 may have an essential role in this complex computational task. Here we show, however, that the spatial selectivity exhibited by the large majority of A1 neurons is well predicted by a simple linear model, which assumes that neurons additively integrate sound levels in each frequency band and ear. The success of this linear model is surprising, given that computing sound source direction is a necessarily nonlinear operation. However, because linear operations preserve information, our results are consistent with the hypothesis that A1 may also form a gateway to higher, more specialized cortical areas.     

Columns for visual features of objects in monkey inferotemporal cortex.

At early stages of the mammalian visual cortex, neurons with similar stimulus selectivities are vertically arrayed through the thickness of the cortical sheet and clustered in patches or bands across the surface. This organization, referred to as a 'column', has been found with respect to one-dimensional stimulus parameters such as orientation of stimulus contours, eye dominance of visual inputs, and direction of stimulus motion. It is unclear, however, whether information with extremely high dimensions, such as visual shape, is organized in a similar columnar fashion or in a different manner in the brain. Here we report that the anterior inferotemporal area of the monkey cortex, the final station of the visual cortical stream crucial for object recognition, consists of columns, each containing cells responsive to similar visual features of objects.     

Functional specialisation in the visual cortex of the rhesus monkey.

Anatomical and functional studies of the visual cortex of the rhesus monkey have shown that it is made up of a multiplicity of distinct areas. These seem to be functionally specialised to analyse different features of the visual environment.     

Foci of orientation plasticity in visual cortex

Cortical areas are generally assumed to be uniform in their capacity for adaptive changes or plasticity. Here we demonstrate, however, that neurons in the cat striate cortex (V1) show pronounced adaptation-induced short-term plasticity of orientation tuning primarily at specific foci. V1 neurons are clustered according to their orientation preference in iso-orientation domains that converge at singularities or pinwheel centres. Although neurons in pinwheel centres have similar orientation tuning and responses to those in iso-orientation domains, we find that they differ markedly in their capacity for adaptive changes. Adaptation with an oriented drifting grating stimulus alters responses of neurons located at and near pinwheel centres to a broad range of orientations, causing repulsive shifts in orientation preference and changes in response magnitude. In contrast, neurons located in iso-orientation domains show minimal changes in their tuning properties after adaptation. The anisotropy of adaptation-induced orientation plasticity is probably mediated by inhomogeneities in local intracortical interactions that are overlaid on the map of orientation preference in V1.     

Electrophysiological measurement of rapid shifts of attention during visual search.

The perception of natural visual scenes that contain many objects poses computational problems that are absent when objects are perceived in isolation. Vision researchers have captured this attribute of real-world perception in the laboratory by using visual search tasks, in which subjects search for a target object in arrays containing varying numbers of non-target distractor objects. Under many conditions, the amount of time required to detect a visual search target increases as the number of objects in the stimulus array increases, and some investigators have proposed that this reflects the serial application of attention to the individual objects in the array. However, other investigators have argued that this pattern of results may instead be due to limitations in the processing capacity of a parallel processing system that identifies multiple objects concurrently. Here we attempt to address this longstanding controversy by using an electrophysiological marker of the moment-by-moment direction of attention-the N2pc component of the event-related potential waveform--to show that attention shifts rapidly among objects during visual search.