As we know, nuclear norm based regularization methods have the real-world applications in pattern recognition and computer vision. However, there exists a biased estimator when nuclear norm relaxes the rank function. To solve this issue, we focus on studying nonconvex rank regularization problems for both robust matrix completion (RMC) and low rank representation (LRR), respectively. By extending both to a general low rank matrix minimization problem, we develop a nonconvex alternating direction method of multipliers (ADMM). Moreover, the convergence results, i.e., the variable sequence generated by the nonconvex ADMM is bounded and its subsequence converges to a stationary point. Meanwhile, its limiting point satisfies the Karush-Kuhn-Tucher (KKT) conditions provided under some milder assumptions. Numerical experiments can verify the convergence properties of the theoretical results and the performance shows its superiority on both image inpainting and subspace clustering.