The block-based analysis for tamper localization is a prevailing mechanism in hash-based forgery detection algorithm. One of the main problems with a block-based analysis is its rough localization stemming from the demand to use relatively large blocks to reduce hash length. While decreasing the block size can improve the localization resolution, the hash length tends to become too long to be practical. In this paper, we propose a binary ranking hashing approach that satisfies both the requirements of compact hash length and small block size, to obtain a binary map combined with spatial information. Meanwhile, we investigate a multiscale difference map fusion approach that fuses multiple candidate difference maps, resulting from the analysis of the subtraction between two binary maps with different sliding windows, to obtain a single, more reliable tampering map with better localization resolution. We use manifold ranking to model this multiscale difference map fusion problem and propose a two-stage scheme, namely, ranking with tampering queries and nontampering queries. Our results indicate that the proposed tamper detection method can improve the tamper localization resolution compared with state-of-the-art methods.