Zero-shot image retrieval involves retrieving images of unseen classes using a query image of the same class. To determine whether a given image is of the same class as the query image, a universal threshold of similarity measures is needed, as class-specific thresholds are not feasible for unseen classes. However, existing methods for zero-shot image retrieval focus on pushing a margin between intra-class and inter-class similarities for each class during the training phase. This approach can result in varying decision boundaries between intra- and inter-class similarities across classes, which could compromise performance when a universal threshold is used in the inference stage. Additionally, for classes with low intra-class variances or inter-class correlations, the pushing force of the margin-pushing approach might be too weak to learn high-quality embeddings. To address these issues, we propose a novel Consistent Penalizing Field (CPF) Loss for zero-shot image retrieval. The proposed method has a single consistent penalizing field for all classes, resulting in similar decision boundaries across classes. By penalizing samples outside the penalizing field, CPF Loss can better utilize the information of samples with highly unbalanced intra-class and inter-class correlations, and improve the discriminative power of DML learning for zero-shot image retrieval. Extensive experiments are conducted on the challenging Shopee Product Matching dataset and other established benchmarks, and the results demonstrate that the proposed method consistently outperforms the state-of-the-art methods. The code is available at https://github.com/cloudlc/CPF.