As one of the most successful feature extraction algorithms, scale invariant feature transform( SIFT) has been widely employed in many applications. Recently, the security of SIFT against malicious attack has been attracting increasing attention, and several techniques have been devised to remove SIFT keypoints intentionally. However, most of the existing methods still suffer from the following three problems: 1) the keypoint removal rate achieved by many techniques is unsatisfactory when removing keypoints within multiple octaves; 2) noticeable artifacts are introduced in the processed image, especially in those highly textured regions; and 3) the color information is totally neglected, precluding the widespread adoption of those methods. To tackle these challenges, in this paper, we propose a novel SIFT keypoint removal framework. By modeling the difference of Gaussian space as a directed weighted graph, we derive a set of strict inequality constraints to remove a SIFT keypoint along a pre-constructed acyclic path. To minimize the incurred distortion, the path is strategically designed over the directed graph. Furthermore, we propose a simple yet effective optimization framework for recovering the color information of the keypoint-removed image. Extensive experiments are provided to show the superior performance of our proposed scheme over the state-of-the-art techniques, in both the scenarios of removing keypoints in a single octave and in multiple octaves.