In this paper, we study the numerical solution for the time-fractional advection-diffusion equation (TFADE). A high-order accurate three-point combined compact difference scheme with the L1 formula is proposed to solve a class of TFADEs. The method is globally (2-\gamma)th-order accurate in time and at least fifth-order accurate in space for the constant coefficient TFADEs subject to periodic boundary conditions, where $\gamma$ is the order of time-fractional derivative in the governing equation. The unconditional stability and the high-order convergence are proved using the Fourier analysis method. The proposed method can be extended to treat the variable coefficient TFADEs subject to other local boundary conditions. Several numerical examples are computed to validate the numerical accuracy, effectiveness and robustness of the present method for TFADEs with smooth solutions} and different boundary conditions. Moreover, the proposed algorithm preserves the higher order accuracy for advection-dominated problems with the smooth solutions.