This paper proposes a disturbance recognition method for phase-sensitive optical time-domain reflectometry (F-OTDR) based on Faster-RCNN. The method achieves high-speed detection of intrusion location and classification with high accuracy. Our scheme makes full use of the 2D sensing information on spatial-temporal images and uses the advanced "two-step" object detection algorithm Faster-RCNN to achieve real-time operation. Firstly, to improve the detection speed, Region Proposal Network (RPN) and Region of Interest (RoI) are used. Secondly, our CNN-based approach can extract features automatically of disturbance events from spatial-temporal images. So, it has better robustness compared to traditional machine learning methods. Thirdly, the method uses an end-to-end CNN object detection model that integrates multiple modules into a single network. Therefore, it has a significant advantage in detection speed. We conducted data collection under perimeter security scenarios and acquired 4 types of events with a total of 4987 samples. The four events contain “rigid collision”, “hitting net”, “shaking net”, and “cutting net”, which are representative in the perimeter security scenario. Experimental results proves that our method can achieve a real-time operation (0.1659 s processing time for 0.5 s sensing data) with high accuracy (96.32%), shows great potential in real-time disturbance detection for online monitoring industrial application of F-OTDR.