Physiological tremor forecasting is one of the most important issues in tele-operation that can improve the operational precision greatly. In the tele-operation, signals are three-dimensional and nonstationary time series. In this paper, a wavelet broad learning adaptive filter (WBLAF)is proposed to forecast and cancel the physiological tremor in tele-operation. Firstly, the variant architecture of original broad learning system (BLS)is proposed to extract the features of physiological tremor and the coupling information between dimensions and the novel WBLAF maps each dimensional data as the feature nodes respectively by the wavelet function. Secondly, in order to make the system can use the advantage of wavelet to extract the time-frequency features of input, a novel self-paced wavelet auto-encoder (SPWAE)is proposed to train the weights of feature mapping. Moreover, the ridge regression learning algorithm and the incremental learning of the proposed filter are applied for online learning. Finally, we designed three parts of experiments to show its effectiveness and feasibility. The first numerical experiment shows the convergence of the novel SPWAE and the second numerical experiment verifies the ability of WBLAF to predict non-stationary time series signals. In the end, semi-physical simulation experiment is implementation. As shown in the results, the novel method can effectively predict and filter out the physiological tremor in tele-operation.