In this work, we propose a novel clustering framework by gradually shrinking the graph of samples called adaptive graph shrinking (AGS). It is motivated by the smoothness of graph signal which will reach a lower bound when samples from the same cluster merge into one component of a graph. We mimic the merging process by using some dynamic clients to represent original samples. The dynamic nature of representatives also reduces to a dynamic graph which endows the final stable graph a lower smoothness, whereas the previous work robust continuous clustering (RCC) uses a fixed graph. This dynamic process is realized by alternatively optimizing the representatives and weights of the graph. We perform experiments on two public database COIL20 and MNIST to demonstrate that the dynamically shrinking of the graph is able to promote the clustering performance.