Depth is encoded in the visual cortex by a specialized receptive field structure

Binocular neurons in the visual cortex are thought to perform the first stage of processing for the fine stereoscopic depth discrimination exhibited by animals with frontally located eyes. Because lateral separation of the eyes gives a slightly different view to each eye, there are small variations in position (disparities), mainly along the horizontal dimension, between corresponding features in the two retinal images. The visual system uses these disparities to gauge depth. We studied neurons in the cat's visual cortex to determine whether the visual system uses the anisotropy in the range of horizontal and vertical disparities. We report here that there is a corresponding anisotropy in the cortical representation of binocular information: receptive-field profiles for left and right eyes are matched for cells that are tuned to horizontal orientations of image contours. For neurons tuned to vertical orientations, left and right receptive fields are predominantly dissimilar. Therefore, a major modification is required of the conventional notion of disparity processing. The modified scheme allows a unified encoding of monocular form and binocular disparity information.     

Motion direction, speed and orientation in binocular matching

The spatial differences between the images seen by the two eyes, called binocular disparities, can be used to recover the volumetric (three-dimensional) aspects of a scene. The computation of disparity depends upon the correct identification of corresponding features in the two images. Understanding what image features are used by the brain to solve this matching problem is one of the main issues in stereoscopic vision. Many cortical neurons in visual areas V1 (ref. 2), MT (refs 3, 4) and MST (refs 5, 6) that are tuned to binocular disparity are also tuned to orientation, motion direction and speed. Although psychophysical work has shown that motion direction can facilitate binocular matching, the psychophysical literature on the role of orientation is mixed, and it has been argued that speed differences are ineffective in aiding correspondence. Here we use a different psychophysical paradigm to show that the visual system uses similarities in orientation, motion direction and speed to achieve binocular correspondence. These results indicate that cells that multiplex orientation, motion direction, speed and binocular disparity may help to solve the binocular matching problem.     

An unexpected specialization for horizontal disparity in primate primary visual cortex

The horizontal separation of the eyes means that objects nearer or farther than the fixation point project to different locations on the two retinae, differing principally in their horizontal coordinates (horizontal binocular disparity). Disparity-selective neurons have generally been studied with disparities applied in only one direction (often horizontal), which cannot determine whether the encoding is specialized for processing disparities along the horizontal axis. It is therefore unclear if disparity selectivity represents a specialization for naturally occurring disparities. I used random dot stereograms to study disparity-selective neurons from the primary visual cortex (V1) of awake fixating monkeys. Many combinations of vertical and horizontal disparity were used, characterizing the surface of responses as a function of two-dimensional disparity. Here I report that the response surface usually showed elongation along the horizontal disparity axis, despite the isotropic stimulus. Thus these neurons modulated their firing rate over a wider range of horizontal disparity than vertical disparity. This demonstrates that disparity-selective cells are specialized for processing horizontal disparity, and that existing models of disparity selectivity require substantial revision.     

对立体视觉中对称性假设的检验

指出了基于立体视觉研究中的隐性对称性假设,即认为匹配的选取与哪只眼观察哪一图像无关,而仅取决于两眼映像特征的相似性,因此交换左眼和右眼图像这样一种实验操作,被认为提供了一种非常简便有效的技术,用于鉴定双眼性视差和其他不同于双眼性视差的因素对立体深度的贡献.采用2:2直线型常规立体视构型,对比了左右眼图像交换前后的匹配结果,发现匹配融合产生中间第3根直线的两根直线在图像交换前后会发生改变,并导致中间直线的深度在图像交换前后不发生颠倒.实验结果证明:立体视觉研究中的隐性对称性假设不成立;交换左眼和右眼图像后深度是否颠倒,并不是判断深度是来源于匹配视差还是其他因素的依据.     

基于双目立体视觉的列车目标识别和测距技术

随着近几年铁路运输量的大幅度提升,铁路运输自动化被提上了日程,现如今火车的调车与编组的效率都有了大幅度提高,但是,火车的摘钩分解操作依旧是需要人工完成.针对摘钩作业时间短,需要在机器人和车厢同步时准确识别目标把手和测距的问题,提出使用双目立体视觉技术配合机械臂自动摘取车钩的方法并对视觉部分进行深入研究:通过图像预处理和模板匹配技术识别目标把手,在利用特征检测和匹配算法恢复双目摄像机间的位姿信息,并对双目摄像机的位姿信息进行校正.为了克服立体匹配时光照不均的影响,提出了基于局部融合的立体匹配算法获取视差图,最后使用三角测量计算列车分解区域中目标把手的深度信息,实现三维重建.此方法可以在识别把手位置后,测量把手距双目视觉系统的物理距离,为机器人自动摘钩提供数据基础.     

Coding of smooth eye movements in three-dimensional space by frontal cortex

Through the development of a high-acuity fovea, primates with frontal eyes have acquired the ability to use binocular eye movements to track small objects moving in space. The smooth-pursuit system moves both eyes in the same direction to track movement in the frontal plane (frontal pursuit), whereas the vergence system moves left and right eyes in opposite directions to track targets moving towards or away from the observer (vergence tracking). In the cerebral cortex and brainstem, signals related to vergence eye movements--and the retinal disparity and blur signals that elicit them--are coded independently of signals related to frontal pursuit. Here we show that these types of signal are represented in a completely different way in the smooth-pursuit region of the frontal eye fields. Neurons of the frontal eye field modulate strongly during both frontal pursuit and vergence tracking, which results in three-dimensional cartesian representations of eye movements. We propose that the brain creates this distinctly different intermediate representation to allow these neurons to function as part of a system that enables primates to track and manipulate objects moving in three-dimensional space.     

Correlated binocular activity guides recovery from monocular deprivation

Monocular deprivation (MD) has much more rapid and severe effects on the ocular dominance of neurons in the primary visual cortex (V1) than does binocular deprivation. This finding underlies the widely held hypothesis that the developmental plasticity of ocular dominance reflects competitive interactions for synaptic space between inputs from the two eyes. According to this view, the relative levels of evoked activity in afferents representing the two eyes determine functional changes in response to altered visual experience. However, if the deprived eye of a monocularly deprived kitten is simply reopened, there is substantial physiological and behavioural recovery, leading to the suggestion that absolute activity levels, or some other non-competitive mechanisms, determine the degree of recovery from MD. Here we provide evidence that correlated binocular input is essential for such recovery. Recovery is far less complete if the two eyes are misaligned after a period of MD. This is a powerful demonstration of the importance of cooperative, associative mechanisms in the developing visual cortex.     

Disparity curvature and the perception of three-dimensional surfaces

BINOCULAR stereopsis provides information about the relative distance of objects from the differences in the horizontal position of their images on the two retinas. Because the size of the disparity between two points is inversely related to the square of the viewing distance, it is usually assumed that disparities have to be scaled according to distance using the vergence angle of the eyes, or by using the small vertical disparities that also exist between corresponding points of the two images. Here we present evidence that the visual system could extract information about the shapes of surfaces (without the need for scaling) by using the second spatial derivative of disparity--disparity curvature--which remains invariant with viewing distance. Rather than computing the second derivative, we suggest that an approximation to disparity curvature could be derived from the differences in curvature of corresponding line elements in the two eyes.     

Interpolation in stereoscopic matching

Anyone who has stared at a repeating wallpaper pattern, or a periodic pattern of tiles, has probably experienced the phenomenon of a false stereoscopic depth percept. This arises because of a mismatching in the two eyes of repeating elements in the pattern. The phenomenon is less likely to occur if an edge of the textured region is in view; the edge seems to fix the registration of elements. We describe here a stereogram which exemplifies this principle; it has a central, periodic region bounded on either side by edges with pre-assigned disparities. We find that the perceived depth of the central region is controlled by the edges. In certain conditions (when the period is spatially large), the edges simply impose one of the expected discrete matchings. In other conditions, however, we observe a striking phenomenon: interpolation in depth occurs between the edges, violating any possible feature-by-feature matching.     

基于多尺度的小基高比立体匹配方法

针对小基高比立体匹配中的＂黏合＂现象和深度精度问题,提出一种小基高比立体匹配方法.该方法通过将自适应窗口技术和多窗口策略相结合为参考图像确定匹配窗口;然后根据规范化互相关函数和＂胜者全取＂策略计算整数级视差;再以整数级视差为基础利用基于二分法的亚像素匹配方法计算亚像级视差;最后采用基于图像分割的迭代传播方法以获得稠密视差图.实验结果表明：该立体匹配算法减少了小基高比匹配中的＂黏合＂现象,同时获得了稠密的高精度亚像素级视差,其亚像素精度可优于1/20个像元.     

A neural representation of depth from motion parallax in macaque visual cortex

Perception of depth is a fundamental challenge for the visual system, particularly for observers moving through their environment. The brain makes use of multiple visual cues to reconstruct the three-dimensional structure of a scene. One potent cue, motion parallax, frequently arises during translation of the observer because the images of objects at different distances move across the retina with different velocities. Human psychophysical studies have demonstrated that motion parallax can be a powerful depth cue, and motion parallax seems to be heavily exploited by animal species that lack highly developed binocular vision. However, little is known about the neural mechanisms that underlie this capacity. Here we show, by using a virtual-reality system to translate macaque monkeys (Macaca mulatta) while they viewed motion parallax displays that simulated objects at different depths, that many neurons in the middle temporal area (area MT) signal the sign of depth (near versus far) from motion parallax in the absence of other depth cues. To achieve this, neurons must combine visual motion with extra-retinal (non-visual) signals related to the animal's movement. Our findings suggest a new neural substrate for depth perception and demonstrate a robust interaction of visual and non-visual cues in area MT. Combined with previous studies that implicate area MT in depth perception based on binocular disparities, our results suggest that area MT contains a more general representation of three-dimensional space that makes use of multiple cues.     

双通道立体图像质量评价方法的研究

针对立体图像质量评价问题,基于人眼观测图像的感知特性,提出一种双通道立体图像质量评价算法。首先,获取双目视图的拉普拉斯金字塔序列构建融合图,采用并行域分解多权重化策略提取双目局部质量感知特征;然后,结合视觉平衡特性引入语义特征通道提取双目高层次语义特征;最后,在支持向量回归的基础上得到双通道主客观图像质量评价值的关系映射。双通道网络集成了包含视差信息的多局部细节特征与全局语义特征,在LIVE 3D 立体图像库进行性能测试,结果表明,算法所得预测值与主观评价值间具有良好的一致性。     

Monocular aniseikonia: a motion parallax analogue of the disparity-induced effect

Mayhew and Longuet-Higgins have recently outlined a computational model of binocular depth perception in which the small vertical disparities between the two eyes' views of a three-dimensional scene are used to determine the 'viewing parameters' of fixation distance (d) and the angle of asymmetric convergence of the eyes (g). The d/g hypothesis, as it has been called, correctly predicts that a fronto-parallel surface, viewed with a vertically magnifying lens over one eye, should appear to be rotated in depth about a vertical axis. We report here a comparable illusion for surfaces specified by monocular motion parallax information, which can be explained more simply by considering the differential invariants of the optic flow field. In addition, our observations suggest that the disparity-induced effect is not a 'whole field' phenomenon nor one limited to small magnification differences between the eyes.     

Gaze direction controls response gain in primary visual-cortex neurons

To localize objects in space, the brain needs to combine information about the position of the stimulus on the retinae with information about the location of the eyes in their orbits. Interaction between these two types of information occurs in several cortical areas, but the role of the primary visual cortex (area V1) in this process has remained unclear. Here we show that, for half the cells recorded in area V1 of behaving monkeys, the classically described visual responses are strongly modulated by gaze direction. Specifically, we find that selectivity for horizontal retinal disparity-the difference in the position of a stimulus on each retina which relates to relative object distance-and for stimulus orientation may be present at a given gaze direction, but be absent or poorly expressed at another direction. Shifts in preferred disparity also occurred in several neurons. These neural changes were most often present at the beginning of the visual response, suggesting a feedforward gain control by eye position signals. Cortical neural processes for encoding information about the three-dimensional position of a stimulus in space therefore start as early as area V1.     

基于双棱镜单摄像机立体视觉的P-GMAW熔池表面重建

利用传统的双目立体视觉技术进行焊接熔池表面形貌的三维传感,系统成本昂贵,且拍摄时必须保证同步,摄像机参数的不一致也容易造成重建误差。分析了双棱镜立体视觉系统原理,搭建了一套基于双棱镜的单摄像机立体视觉系统,在一个CCD像面上拍摄到同一物体存在视差的两幅图像,预处理后通过SSD算法求取像素级视差图,然后由二次曲线拟合进一步获得亚像素级视差图,由标定结果求得每个图像点对应的三维坐标。通过搭建的传感系统获得了P-GMAW平板堆焊基值期间的熔池图像,依照上述流程重建了熔池的三维表面形貌,同时针对重建结果做了误差分析,证实了其可靠性。     

Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus

When dissimilar images are presented to the two eyes, they compete for perceptual dominance so that each image is visible in turn for a few seconds while the other is suppressed. Such binocular rivalry is associated with relative suppression of local, eye-based representations that can also be modulated by high-level influences such as perceptual grouping. However, it is currently unclear how early in visual processing the suppression of eye-based signals can occur. Here we use high-resolution functional magnetic resonance imaging (fMRI) in conjunction with a new binocular rivalry stimulus to show that signals recorded from the human lateral geniculate nucleus (LGN) exhibit eye-specific suppression during rivalry. Regions of the LGN that show strong eye-preference independently show strongly reduced activity during binocular rivalry when the stimulus presented in their preferred eye is perceptually suppressed. The human LGN is thus the earliest stage of visual processing that reflects eye-specific dominance and suppression.     

Sensitivity for vertical retinal image differences

The detection of horizontal disparities between the right and left retinal images is accepted as an important component in the visual perception of depth. It has occasionally been pointed out that a disparity will also occur in the vertical when a target is nearer to one eye than the other and could be used as a second component. Here it is shown that while it is possible to detect size differences between the retinal images in the vertical direction, the sensitivity is at least one order of magnitude less than the horizontal differences. It had been generally accepted that vertical disparities cannot be detected (literature summarized by Tschermak), but there are clear reports of the ability of subjects to respond to relative differences in the vertical magnification of the images of the two eyes, the so-called induced size effect. It manifests itself in the appearance of a visual scene as if there had been an opposite horizontal magnification and is, therefore, in the correct direction if it is to be utilized in asymmetrical convergence.     

The motor side of depth vision

To achieve stereoscopic vision, the brain must search for corresponding image features on the two retinas. As long as the eyes stay still, corresponding features are confined to narrow bands called epipolar lines. But when the eyes change position, the epipolar lines migrate on the retinas. To find the matching features, the brain must either search different retinal bands depending on current eye position, or search retina-fixed zones that are large enough to cover all usual locations of the epipolar lines. Here we show, using a new type of stereogram in which the depth image vanishes at certain gaze elevations, that the search zones are retina-fixed. This being the case, motor control acquires a crucial function in depth vision: we show that the eyes twist about their lines of sight in a way that reduces the motion of the epipolar lines, allowing stereopsis to get by with smaller search zones and thereby lightening its computational load.     

机器人双目图像实时立体匹配算法

针对移动机器人平台上的双目视觉深度信息获取问题,研究了一种适用于动态图像序列的双目图像立体匹配算法.采用成熟的半全局立体匹配算法对图像快速立体匹配;分析了在复杂环境下的运动平台上完成立体匹配所面临的主要噪声干扰,包括白斑噪声和断层闪烁噪声;为此,分别提出了白斑滤波算法和视差图时间域滤波方法以抑制这两种噪声.试验结果表明,所提出的算法能够有效抑制动态双目图像在立体匹配时的噪声干扰,最终能够实时获得连续清晰的视差图序列.      

影响立体图像舒适度的饱和度范围测定

针对立体成像领域的视觉舒适度问题,结合人眼视觉系统的特性定量研究了饱和度因素对观看双目立体图像舒适度的影响．采用三级步长逼近法,通过大量主观实验定义了双目立体图像舒适饱和度匹配图和舒适饱和度差异图,得到了立体图像饱和度匹配范围,并进一步验证了两图的普适性,根据两图判定立体图像是否舒适的准确率达到92％．实验结果能够为立体图像舒适度的主客观评价提供依据．