Open set domain adaptation aims to transfer knowledge in the presence of unknown samples in the target domain. Previous approaches use additional classifiers or threshold-based methods to identify unknown samples and try to investigate the information of class diversity within the unknown samples. Despite achieving excellent adaptation results, these methods ignore those samples that lie on the cluster boundaries, especially the clustering-based methods. In this paper, we propose a novel Neighbor Prototype Contrastive Clustering (NPC) method, which uses the Local Semantic Structure (LSS) to help these low-confidence samples located on the boundary of clusters to return to their own clusters. Further, we propose Local Semantic Consistency (LSC) to evaluate the clustering result and apply it to the domain adaptation process as a metric to assess the reliability of the samples. Results on four benchmarks show that our NPC significantly outperforms most state-of-the-art methods with higher LSC.