Deep semi-nonnegative matrix factorization can obtain the hidden hierarchical representations according to the unknown attributes of the given data. On the other hand, the inherent structure of the each data cluster can be described by the distribution of the intra-class data. Then one hopes to learn a new low dimensional representation which can preserve the intrinsic structure embedded in the original high dimensional data space perfectly. Here we propose a novel distribution preserving deep semi-nonnegative matrix factorization method (DPNMF) to achieve this goal. As a result, the manifold structures in the raw data are well preserved in the feature space being from the top layer. The experimental results on the real-world datasets show that the proposed algorithm has good performance in terms of cluster accuracy and normalized mutual information (NMI).