This paper presents a robust filter called the quaternion Hardy filter (QHF) for color image edge detection. The QHF can be capable of color edge feature enhancement and noise resistance. QHF can be used flexibly by selecting suitable parameters to handle different levels of noise. In particular, the quaternion analytic signal, which is an effective tool in color image processing, can also be produced by quaternion Hardy filtering with specific parameters. Based on the QHF and the improved Di Zenzo gradient operator, a novel color edge detection algorithm is proposed; importantly, it can be efficiently implemented by using the fast discrete quaternion Fourier transform technique. From the experimental results, we conclude that the minimum PSNR improvement rate is 2.3% and the minimum SSIM improvement rate is 30.2% on the CSEE database. The experiments demonstrate that the proposed algorithm outperforms several widely used algorithms.