Aiming at the low accuracy problem of insulator fault detection caused by the fault position occupies a small proportion in the image and complex background environment in aerial images, an insulator fault detection method based on optimized Cascade R-CNN model is proposed in this paper. Based on the original Cascade R-CNN model, deformable convolution is inserted into the backbone network to learn geometric transformation capabilities, and balance loss function is introduced in the detector to balance difficult and easy samples. In the model training phase, the faulty insulator samples are enriched by using Copy-Paste and Mosica, and the positive and negative samples are balanced. The proposed model is tested for insulator fault detection. Compared with the traditional Cascade R-CNN model, the average recall of the optimized loss function model improves 0.38%. Comparing with the Faster R-CNN model, the average recall of the Cascade R-CNN model after introducing variable convolution improves from 89.78% to 93.49%. The results indicate that the proposed model can overcome the interference of samples shielding and sample imbalance effectively.