The convolutional neural network algorithm is widely applied in the automatic inspection of transmission lines. However, the generalization ability of traditional convolutional neural network power defect-recognition model is not ideal. Under the background, this paper proposes a cross-view relation region convolutional neural network (CVR-RCNN) detection algorithm that integrates dual-angle image information, which utilizes two-view visible light images to identify typical defects in transmission lines. The testing shows that the CVR-RCNN model has good robustness. The area under curve (AUC) value of the receiver operating characteristic (ROC) curve is as high as 0.927, and the defect detection accuracy is significantly improved compared with traditional algorithms. Therefore, CVR-RCNN can significantly improve power defect detection and improve the accuracy and stability of the algorithm architecture for the automatic inspection of transmission lines by UAVs.