In this paper, we propose a fast second-order approximation to the variable-order (VO) Caputo fractional derivative, which is developed based on L2-1 formula and the exponential-sum-approximation technique. The fast evaluation method can achieve the second-order accuracy and further reduce the computational cost and the acting memory for the VO Caputo fractional derivative. This fast algorithm is applied to construct a relevant fast temporal second-order and spatial fourth-order scheme (F L2-1 scheme) for the multi-dimensional VO time-fractional sub-diffusion equations. Theoretically, F L2-1 scheme is proved to fulfill the similar properties of the coefficients as those of the well-studied L2-1 scheme. Therefore, F L2-1 scheme is strictly proved to be unconditionally stable and convergent. A sharp decrease in the computational cost and the acting memory is shown in the numerical examples to demonstrate the efficiency of the proposed method.