In this paper, a new learned basis set algorithm (3D-LBT) based on 3D-DCT (Discrete Cosine Transform) is proposed for video fingerprinting and matching, in which for different video categories an Adaboost-based machine learning method is applied to each category of videos for selecting suitable sets of 3D-DCT coefficients to generate fingerprints, and a weighted distance of fingerprints is also defined for fingerprint matching. Our experimental results have illustrated that the proposed algorithm outperforms the conventional 3D-DCT algorithm and the 3D-RBT (Randomized Basis seT) algorithm in terms of robustness and uniqueness. Moreover, the proposed algorithm has better security performance for copyright applications.