融合物体空间关系机制的图像摘要生成方法

聚焦于图像中物体间位置关系这一特定信息,提出一种融合空间关系机制的神经网络图像摘要生成模型,以期为视觉问答和语音导航等下游任务提供物体方位或轨迹等关键信息.为了增强图像编码器的物体间位置关系学习能力,通过改进Transformer结构来引入几何注意力机制,显式地将物体间位置关系融合进物体外观信息中.为了辅助完成面向特定...

Object Space Relation Mechanism Fused Image Caption Method

Focusing on the specific information of the positional relationship between objects in the image, a neural network image summary generation model integrating spatial relationship mechanism is proposed, in order to provide key information (object position or trajectory) for downstream tasks such as visual question answering and voice navigation. In order to enhance the learning ability of the positional relationship between objects of the image encoder, the geometric attention mechanism is introduced by improving the Transformer structure, and the positional relationship between objects is explicitly integrated into the appearance information of the objects. In order to assist in the completion of specific information-oriented extraction and summary generation tasks, a data production method for relative position relations is further proposed, and the image abstract data set Re-Position of the position relations between objects is produced based on the SpatialSense data set. The experimental results of comparative evaluation with five typical models show that the five indicators of the proposed model are better than those of other models on the public test set COCO, and all six indicators are better than those of other models on Re-Position data set.