Deep learning has been widely applied to vision-based structural damage detection, but its computational demand is high. To avoid this computational burden, a novel crack detection system, namely, fusion features-based broad learning system (FF-BLS), is proposed for efficient training without GPU acceleration. In FF-BLS, a convolution module with fixed weights is used to extract the fusion features of images. Feature nodes and enhancement nodes randomly generated by fusion features are used to estimate the output of the network. Meanwhile, the proposed FF-BLS is a dynamical system, which achieves incremental learning by adding nodes. Thus, the trained FF-BLS model can be updated efficiently with additional data, and this substantially reduces the training cost. Finally, FF-BLS was applied to crack detection. Compared with some well-known deep convolutional neural networks (VGG16, ResNet50, InceptionV3, Xception, and EfficientNet), the FF-BLS achieved a similar level of recognition accuracy, but the training speed was increased by more than 20 times.