Miniature eye movements enhance fine spatial detail

Our eyes are constantly in motion. Even during visual fixation, small eye movements continually jitter the location of gaze. It is known that visual percepts tend to fade when retinal image motion is eliminated in the laboratory. However, it has long been debated whether, during natural viewing, fixational eye movements have functions in addition to preventing the visual scene from fading. In this study, we analysed the influence in humans of fixational eye movements on the discrimination of gratings masked by noise that has a power spectrum similar to that of natural images. Using a new method of retinal image stabilization, we selectively eliminated the motion of the retinal image that normally occurs during the intersaccadic intervals of visual fixation. Here we show that fixational eye movements improve discrimination of high spatial frequency stimuli, but not of low spatial frequency stimuli. This improvement originates from the temporal modulations introduced by fixational eye movements in the visual input to the retina, which emphasize the high spatial frequency harmonics of the stimulus. In a natural visual world dominated by low spatial frequencies, fixational eye movements appear to constitute an effective sampling strategy by which the visual system enhances the processing of spatial detail.     

Temporal dynamics of a neural solution to the aperture problem in visual area MT of macaque brain

A critical step in the interpretation of the visual world is the integration of the various local motion signals generated by moving objects. This process is complicated by the fact that local velocity measurements can differ depending on contour orientation and spatial position. Specifically, any local motion detector can measure only the component of motion perpendicular to a contour that extends beyond its field of view. This "aperture problem" is particularly relevant to direction-selective neurons early in the visual pathways, where small receptive fields permit only a limited view of a moving object. Here we show that neurons in the middle temporal visual area (known as MT or V5) of the macaque brain reveal a dynamic solution to the aperture problem. MT neurons initially respond primarily to the component of motion perpendicular to a contour's orientation, but over a period of approximately 60 ms the responses gradually shift to encode the true stimulus direction, regardless of orientation. We also report a behavioural correlate of these neural responses: the initial velocity of pursuit eye movements deviates in a direction perpendicular to local contour orientation, suggesting that the earliest neural responses influence the oculomotor response.     

A new perceptual illusion reveals mechanisms of sensory decoding

Perceptual illusions are usually thought to arise from the way sensory signals are encoded by the brain, and indeed are often used to infer the mechanisms of sensory encoding. But perceptual illusions might also result from the way the brain decodes sensory information, reflecting the strategies that optimize performance in particular tasks. In a fine discrimination task, the most accurate information comes from neurons tuned away from the discrimination boundary, and observers seem to use signals from these 'displaced' neurons to optimize their performance. We wondered whether using signals from these neurons might also bias perception. In a fine direction discrimination task using moving random-dot stimuli, we found that observers' perception of the direction of motion is indeed biased away from the boundary. This misperception can be accurately described by a decoding model that preferentially weights signals from neurons whose responses best discriminate those directions. In a coarse discrimination task, to which a different decoding rule applies, the same stimulus is not misperceived, suggesting that the illusion is a direct consequence of the decoding strategy that observers use to make fine perceptual judgments. The subjective experience of motion is therefore not mediated directly by the responses of sensory neurons, but is only developed after the responses of these neurons are decoded.     

A sensory source for motor variation

Suppose that the variability in our movements is caused not by noise in the motor system itself, nor by fluctuations in our intentions or plans, but rather by errors in our sensory estimates of the external parameters that define the appropriate action. For tasks in which precision is at a premium, performance would be optimal if no noise were added in movement planning and execution: motor output would be as accurate as possible given the quality of sensory inputs. Here we use visually guided smooth-pursuit eye movements in primates as a testing ground for this notion of optimality. In response to repeated presentations of identical target motions, nearly 92% of the variance in eye trajectory can be accounted for as a consequence of errors in sensory estimates of the speed, direction and timing of target motion, plus a small background noise that is observed both during eye movements and during fixations. The magnitudes of the inferred sensory errors agree with the observed thresholds for sensory discrimination by perceptual systems, suggesting that the very different neural processes of perception and action are limited by the same sources of noise.     

Attention modulates synchronized neuronal firing in primate somatosensory cortex

A potentially powerful information processing strategy in the brain is to take advantage of the temporal structure of neuronal spike trains. An increase in synchrony within the neural representation of an object or location increases the efficacy of that neural representation at the next synaptic stage in the brain; thus, increasing synchrony is a candidate for the neural correlate of attentional selection. We investigated the synchronous firing of pairs of neurons in the secondary somatosensory cortex (SII) of three monkeys trained to switch attention between a visual task and a tactile discrimination task. We found that most neuron pairs in SII cortex fired synchronously and, furthermore, that the degree of synchrony was affected by the monkey's attentional state. In the monkey performing the most difficult task, 35% of neuron pairs that fired synchronously changed their degree of synchrony when the monkey switched attention between the tactile and visual tasks. Synchrony increased in 80% and decreased in 20% of neuron pairs affected by attention.     

Neuronal correlates of a perceptual decision

The relationship between neuronal activity and psychophysical judgement has long been of interest to students of sensory processing. Previous analyses of this problem have compared the performance of human or animal observers in detection or discrimination tasks with the signals carried by individual neurons, but have been hampered because neuronal and perceptual data were not obtained at the same time and under the same conditions. We have now measured the performance of monkeys and of visual cortical neurons while the animals performed a psychophysical task well matched to the properties of the neurons under study. Here we report that the reliability and sensitivity of most neurons on this task equalled or exceeded that of the monkeys. We therefore suggest that under our conditions, psychophysical judgements could be based on the activity of a relatively small number of neurons.     

Modulation of the motion aftereffect by selective attention

The motion aftereffect is a much studied and well documented phenomenon. After viewing a moving visual pattern for a period of time, the same pattern appears to drift in the opposite direction when it is stopped. Psychophysical experiments involving interocular transfer, dichoptic stimulation, and motion aftereffects contingent upon other visual parameters such as colour, orientation and texture, imply that the motion aftereffect is generated at the level of the visual cortex. It has been hypothesized that cortical neurons specialized for the detection of motion along a particular direction become 'fatigued' during the adaptation period so that the resting equilibrium subsequently shifts in the opposite direction to that of the adapting stimulus, giving rise to the sensation of the aftereffect. I have found that if observers are engaged in a separate discrimination task superimposed on a moving textured background, the subsequent motion aftereffect to the background is considerably reduced. It seems that motion aftereffects are susceptible to attentional mechanisms.     

Segregation of object and background motion in the retina

An important task in vision is to detect objects moving within a stationary scene. During normal viewing this is complicated by the presence of eye movements that continually scan the image across the retina, even during fixation. To detect moving objects, the brain must distinguish local motion within the scene from the global retinal image drift due to fixational eye movements. We have found that this process begins in the retina: a subset of retinal ganglion cells responds to motion in the receptive field centre, but only if the wider surround moves with a different trajectory. This selectivity for differential motion is independent of direction, and can be explained by a model of retinal circuitry that invokes pooling over nonlinear interneurons. The suppression by global image motion is probably mediated by polyaxonal, wide-field amacrine cells with transient responses. We show how a population of ganglion cells selective for differential motion can rapidly flag moving objects, and even segregate multiple moving objects.     

The perception of heading during eye movements.

When a person walks through a rigid environment while holding eyes and head fixed, the pattern of retinal motion flows radially away from a point, the focus of expansion (Fig. 1a). Under such conditions of translation, heading corresponds to the focus of expansion and people identify it readily. But when making an eye/head movement to track an object off to the side, retinal motion is no longer radial (Fig. 1b). Heading perception in such situations has been modelled in two ways. Extra-retinal models monitor the velocity of rotational movements through proprioceptive or efference information from the extraocular and neck muscles and use that information to discount rotation effects. Retinal-image models determine (and eliminate) rotational components from the retinal image alone. These models have been tested by measuring heading perception under two conditions. First, observers judged heading while tracking a point on a simulated ground plane. Second, they fixated a stationary point and the flow field simulated the effects of a tracking eye movement. Extra-retinal models predict poorer performance in the simulated condition because the eyes do not move. Retinal-image models predict no difference in performance because the two conditions produce identical patterns of retinal motion. Warren and Hannon observed similar performance and concluded that people do not require extra-retinal information to judge heading with eye/head movements present, but they used extremely slow tracking eye movements of 0.2-1.2 deg s-1; a moving observer frequently tracks objects at much higher rates (L. Stark, personal communication). Here we examine heading judgements at higher, more typical eye movement velocities and find that people require extra-retinal information about eye position to perceive heading accurately under many viewing conditions.     

Regulation of the gain of visually guided smooth-pursuit eye movements by frontal cortex

In studies of the neural mechanisms giving rise to behaviour, changes in the neural and behavioural responses produced by a given stimulus have been widely reported. This 'gain control' can boost the responses to sensory inputs that are particularly relevant, select among reflexes for execution by motoneurons or emphasize specific movement targets. Gain control is also an integral part of the smooth-pursuit eye movement system. One signature of gain control is that a brief perturbation of a stationary target during fixation causes tiny eye movements, whereas the same perturbation of a moving target during the active state of accurate pursuit causes large responses. Here we show that electrical stimulation of the smooth-pursuit eye movement region in the arcuate sulcus of the frontal lobe ('the frontal pursuit area', FPA) mimics the active state of pursuit. Such stimulation enhances the response to a brief perturbation of target motion, regardless of the direction of motion. We postulate that the FPA sets the gain of pursuit, thereby participating in target selection for pursuit.     

Dynamic coding of behaviourally relevant stimuli in parietal cortex.

A general function of cerebral cortex is to allow the flexible association of sensory stimuli with specific behaviours. Many neurons in parietal, prefrontal and motor cortical areas are activated both by particular movements and by sensory cues that trigger these movements, suggesting a role in linking sensation to action. For example, neurons in the lateral intraparietal area (LIP) encode both the location of visual stimuli and the direction of saccadic eye movements. LIP is not believed to encode non-spatial stimulus attributes such as colour. Here we investigated whether LIP would encode colour if colour was behaviourally linked to the eye movement. We trained monkeys to make an eye movement in one of two directions based alternately on the colour or location of a visual cue. When cue colour was relevant for directing eye movement, we found a substantial fraction of LIP neurons selective for cue colour. However, when cue location was relevant, colour selectivity was virtually absent in LIP. These results demonstrate that selectivity of cortical neurons can change as a function of the required behaviour.     

Small modulation of ongoing cortical dynamics by sensory input during natural vision

During vision, it is believed that neural activity in the primary visual cortex is predominantly driven by sensory input from the environment. However, visual cortical neurons respond to repeated presentations of the same stimulus with a high degree of variability. Although this variability has been considered to be noise owing to random spontaneous activity within the cortex, recent studies show that spontaneous activity has a highly coherent spatio-temporal structure. This raises the possibility that the pattern of this spontaneous activity may shape neural responses during natural viewing conditions to a larger extent than previously thought. Here, we examine the relationship between spontaneous activity and the response of primary visual cortical neurons to dynamic natural-scene and random-noise film images in awake, freely viewing ferrets from the time of eye opening to maturity. The correspondence between evoked neural activity and the structure of the input signal was weak in young animals, but systematically improved with age. This improvement was linked to a shift in the dynamics of spontaneous activity. At all ages including the mature animal, correlations in spontaneous neural firing were only slightly modified by visual stimulation, irrespective of the sensory input. These results suggest that in both the developing and mature visual cortex, sensory evoked neural activity represents the modulation and triggering of ongoing circuit dynamics by input signals, rather than directly reflecting the structure of the input signal itself.     

Performance monitoring by the supplementary eye field

Intelligent behaviour requires self-control based on the consequences of actions. The countermanding task is designed to study self-control; it requires subjects to withhold planned movements in response to an imperative stop signal, which they can do with varying success. In humans, the medial frontal cortex has been implicated in the supervisory control of action. In monkeys, the supplementary eye field in the dorsomedial frontal cortex is involved in producing eye movements, but its precise function has not been clarified. To investigate the role of the supplementary eye field in the control of eye movements, we recorded neural activity in macaque monkeys trained to perform an eye movement countermanding task. Distinct groups of neurons were active after errors, after successful withholding of a partially prepared movement, or in association with reinforcement. These three forms of activation could not be explained by sensory or motor factors. Our results lead us to put forward the hypothesis that the supplementary eye field contributes to monitoring the context and consequences of eye movements.     

Movement selection in advance of action in the superior colliculus.

The primate superior colliculus contains a map of saccadic eye movements. Saccades are high-velocity eye movements to selected targets in the visual field, but little is known about the neural mechanisms responsible for target selection or the related problem of choosing a particular movement from the oculomotor repertoire. Two classes of neurons have been described in the superior colliculus which show bursts of activity before the saccade: discrete bursters display a vigorous pre-saccadic burst and prelude bursters show low-frequency activity as a prelude to burst onset. We have designed experiments to test whether prelude activity is related to saccade selection. Our tasks use a cue to specify which of two physically identical visual stimuli is the goal of an impending saccade. This cue is spatially and temporally isolated from the potential targets as well as from visual cues signalling movement initiation. Our results show that prelude activity occurs shortly after information is available for correct saccade selection and, more importantly, the activity is predictive of saccade choice. The results thus suggest that the superior colliculus participates in the process of saccade selection.     

Predictable eye-head coordination during driving.

Large changes in the direction of gaze are made with a combination of fast saccadic eye movements and rather slower head movements. Since the first study on freely moving subjects, most authors have agreed that the head movement component of gaze is very variable, with a high 'volitional' component. But in some circumstances head and eye movements can be quite predictable, for example when a subject is asked to shift gaze as quickly as possible. Under these conditions, laboratory studies have shown that the eye and head motor-systems both receive gaze-change commands, although they execute them in rather different ways. Here I reconsider the way gaze direction is changed during free movement, but in the performance of a task where the subject is too busy to exert conscious control over head or eye movements. Using a new portable and inexpensive method for recording head and eye movements, I examine the oculomotor behaviour of car drivers, particularly during the large gaze changes made at road junctions. The results show that the pattern of eye and head movements is highly predictable, given only the sequence of gaze targets.     

The influence of visual motion on fast reaching movements to a stationary object

One of the most important functions of vision is to direct actions to objects. However, every time that vision is used to guide an action, retinal motion signals are produced by the movement of the eye and head as the person looks at the object or by the motion of other objects in the scene. To reach for the object accurately, the visuomotor system must separate information about the position of the stationary target from background retinal motion signals-a long-standing problem that is poorly understood. Here we show that the visuomotor system does not distinguish between these two information sources: when observers made fast reaching movements to a briefly presented stationary target, their hand shifted in a direction consistent with the motion of a distant and unrelated stimulus, a result contrary to most other findings. This can be seen early in the hand's trajectory (approximately 120 ms) and occurs continuously from programming of the movement through to its execution. The visuomotor system might make use of the motion signals arising from eye and head movements to update the positions of targets rapidly and redirect the hand to compensate for body movements.     

Robust judgement of inter-object distance by an arthropod

Animals use several strategies for depth vision, reflecting the constraints imposed by body size, the structure of the visual system and the visual geometry of the environment. Arthropods in particular have restricted depth perception, because they are small and possess closely set, low-resolution compound eyes. Yet, here we show that fiddler crabs defending their burrows from conspecifics can judge how close other crabs are to their burrow. When confronted with small dummy crabs, the burrow owners assess the dummy's position and motion relative to their burrow and not relative to themselves--in other words, by using an allocentric rather than an egocentric frame of reference. Irrespective of their own distance from the dummy, the likelihood that the crabs rush back to defend their burrow increases strongly as the dummy approaches the burrow. In addition, the mean dummy-burrow distance at which the crabs respond is constant and independent of the dummy's direction of approach. We propose that to solve this sophisticated task of relative distance judgement, the crabs combine visual information on dummy position and direction with information on burrow location acquired during path integration. In doing so, the crabs, like humans, make clever use of the visual geometry of their environment.     

Humans use internal models to estimate gravity and linear acceleration

Because sensory systems often provide ambiguous information, neural processes must exist to resolve these ambiguities. It is likely that similar neural processes are used by different sensory systems. For example, many tasks require neural processing to distinguish linear acceleration from gravity, but Einstein's equivalence principle states that all linear accelerometers must measure both linear acceleration and gravity. Here we investigate whether the brain uses internal models, defined as neural systems that mimic physical principles, to help estimate linear acceleration and gravity. Internal models may be used in motor contro, sensorimotor integration and sensory processing, but direct experimental evidence for such models is limited. To determine how humans process ambiguous gravity and linear acceleration cues, subjects were tilted after being rotated at a constant velocity about an Earth-vertical axis. We show that the eye movements evoked by this post-rotational tilt include a response component that compensates for the estimated linear acceleration even when no actual linear acceleration occurs. These measured responses are consistent with our internal model predictions that the nervous system can develop a non-zero estimate of linear acceleration even when no true linear acceleration is present.     

Cortical microstimulation influences perceptual judgements of motion direction

Neurons in the visual cortex respond selectively to perceptually salient features of the visual scene, such as the direction and speed of moving objects, the orientation of local contours, or the colour or relative depth of a visual pattern. It is commonly assumed that the brain constructs its percept of the visual scene from information encoded in the selective responses of such neurons. We have now tested this hypothesis directly by measuring the effect on psychophysical performance of modifying the firing rates of physiologically characterized neurons. We required rhesus monkeys to report the direction of motion in a visual display while we electrically stimulated clusters of directionally selective neurons in the middle temporal visual area (MT, or V5), an extrastriate area that plays a prominent role in the analysis of visual motion information. Microstimulation biased the animals' judgements towards the direction of motion encoded by the stimulated neurons. This result indicates that physiological properties measured at the neuronal level can be causally related to a specific aspect of perceptual performance.     

Motion streaks provide a spatial code for motion direction.

Although many neurons in the primary visual cortex (V1) of primates are direction selective, they provide ambiguous information about the direction of motion of a stimulus. There is evidence that one of the ways in which the visual system resolves this ambiguity is by computing, from the responses of V1 neurons, velocity components in two or more spatial orientations and then combining these velocity components. Here I consider another potential neural mechanism for determining motion direction. When a localized image feature moves fast enough, it should become smeared in space owing to temporal integration in the visual system, creating a spatial signal-a 'motion streak'-oriented in the direction of the motion. The orientation masking and adaptation experiments reported here show that these spatial signals for motion direction exist in the human visual system for feature speeds above about 1 feature width per 100 ms. Computer simulations show that this psychophysical finding is consistent with the known response properties of V1 neurons, and that these spatial signals, when appropriately processed, are sufficient to determine motion direction in natural images.