Broad learning system (BLS) is a fast and efficient learning model. However, BLS has limited representation capacity in the feature mapping layer. Additionally, BLS lacks local mapping capability. To address these problems, a cascaded neural network framework based on a sparse polynomial-based RBF neural network and an attention-based broad learning system (SPRBF-ABLS) is proposed. We first propose a sparse polynomial weight-based RBF neural network (SPRBF) for feature mapping. Then an attention mechanism for BLS is proposed to enhance the representation capacity of BLS. The proposed model is evaluated on regression, classification, and face recognition datasets. In regression and classification experiments, the nonlinear approximation capability of the proposed model outperforms other BLS models. In face recognition experiments, the proposed model can improve the representation capacity, especially the robustness against noisy images. The experiments demonstrate the effectiveness and robustness of the proposed model.