The computation of quaternion singular triplets has become one of the core targets of color image processing. However, the existing algorithms are far from meeting people’s expectations on the computation speed. A novel harmonic multi-symplectic Lanczos algorithm is presented for approximating extreme quaternion singular triplets, which performs real operations entirely and stores only four parts of quaternion matrices or vectors. The underlying theory is to preserve an algebraic structure during the partial bidiagonalization, the argumentation, and the restarted bidiagonalization. Both the smallest and largest quaternion singular triples are computed with high precision and in short calculation time. The proposed algorithm is applied to color video semantic segmentation. Numerical examples on synthetic and color image data sets illustrate that the proposed algorithm is superior to the state-of-the-art algorithms in terms of residual calculation and computational time.