A fast direct method is proposed for the block lower triangular Toeplitz-like with tri-diagonal blocks system which arises from the time-fractional partial differential equation. The proposed method is based on the divide-and-conquer strategy and together with the fast Fourier transforms for calculating Toeplitz matrix-vector multiplication. The complexity needs O(MN log^2 M) arithmetic operations, where M is the number of blocks (the number of time steps) in the system and N is the size (number of spatial grid points) of each block. Numerical examples from the finite difference discretization of time-fractional partial differential equations are given to demonstrate the efficiency of the proposed method.