Aiming at the problems of insufficient use of prior fault information and incomplete fault samples, the existing deep learning methods in the fault diagnosis process of non-stationary rolling bearings have low accuracy or even failure to diagnose. This paper proposed a double branch convolutional neural network method for bearing fault diagnosis based on the research of fault location and damage degree. Firstly, a point transfer matrix was used to process the original vibration signals of rolling bearings, and a two-dimensional image data set was constructed. Secondly, a double branch improved VGGNet deep convolutional network was constructed. One-hot coding of fault location and damage degree separately, so the features and labels of fault location and damage degree can be adaptive simultaneously. The simulation and experimental results show that compared with other neural networks, the proposed method can achieve the multi-state classification of the potential different fault locations and damage degrees of rolling bearings and still maintains good diagnostic performance under the condition of partial lack of prior knowledge.