This paper proposes a real-time multi-class disturbance detection algorithm based on YOLO for distributed fiber vibration sensing. The algorithm achieves real-time detection of event location and classification on external intrusions sensed by distributed optical fiber sensing system (DOFS) based on phase-sensitive optical time-domain reflectometry (Φ-OTDR). We conducted data collection under perimeter security scenarios and acquired five types of events with a total of 5787 samples. The data is used as a spatial–temporal sensing image in the training of our proposed YOLO-based model (You Only Look Once-based method). Our scheme uses the Darknet53 network to simplify the traditional two-step object detection into a one-step process, using one network structure for both event localization and classification, thus improving the detection speed to achieve real-time operation. Compared with the traditional Fast-RCNN (Fast Region-CNN) and Faster-RCNN (Faster Region-CNN) algorithms, our scheme can achieve 22.83 frames per second (FPS) while maintaining high accuracy (96.14%), which is 44.90 times faster than Fast-RCNN and 3.79 times faster than Faster-RCNN. It achieves real-time operation for locating and classifying intrusion events with continuously recorded sensing data. Experimental results have demonstrated that this scheme provides a solution to real-time, multi-class external intrusion events detection and classification for the Φ-OTDR-based DOFS in practical applications.