High-dimensional and large-scale time series processing has aroused considerable research interests during decades. It is difficult for traditional methods to reveal the evolution state in dynamical systems and discover the relationship among variables automatically. In this paper, we propose a unified framework for nonuniform embedding, dynamical system revealing, and time series prediction, termed as Structured Manifold Broad Learning System (SM-BLS). The structured manifold learning is introduced for nonuniform embedding and unsupervised manifold learning simultaneously. Graph embedding and feature selection are both considered to depict the intrinsic structure connections between chaotic time series and its low-dimensional manifold. Compared with traditional methods, the proposed framework could discover potential deterministic evolution information of dynamical systems and make the modeling more interpretable. It provides us a homogeneous way to recover the chaotic attractor from multivariate and heterogeneous time series. Simulation analysis and results show that SM-BLS has advantages in dynamic discovery and feature extraction of large-scale chaotic time series prediction.