Nowadays, the scales of power systems are enlarging, the types of input power sources are increasing, and the energy demands are also raising. Hence, the disturbance in grids become more frequent, which request a more reliable protection relay system. To achieve the timely response for the potential disturbances in protection relay systems, this paper establishes anomaly detection method for warning and analyzing such disturbances. Firstly, the Kernel Linear Discriminant Analysis (KLDA) model is utilized to reduce the dimensionality of input data, thus to decrease the computation burden and accelerate the response. Then, the Influenced Outlierness (INFLO) anomaly detection is designed. This model can find the outliers in time according to the common range of operation setting parameters, and thus to swiftly response to anomaly conditions. Finally, an empirical study which is based on the protection relay system in one operating distribution network is conducted. The results show that the performance of the proposed method is satisfying, and can be deployed to monitor or manage the countermeasures for potential risks.