Anomaly detection for hyperspectral images (HSIs) is a challenging problem to distinguish a few anomalous pixels from a majority of background pixels. Most existing methods cannot simultaneously explore both structural and spatial information from global and local perspectives. In this paper, we propose a stacked graph fusion denoising autoencoder (SGFDAE) for hyperspectral anomaly detection. Specifically, the global and local graphs are constructed from an HSI to explore potential structural and spatial information. With the designed graph fusion strategy, an advanced graph denoising autoencoder with deep architecture is developed in a hierarchical manner. To achieve better reconstruction and detection, a greedy layer-wise unsupervised pre-training strategy is presented for network training. Experiments show that SGFDAE achieves 97.17%, 98.43% and 98.90% detection accuracies by averaging the results of the datasets from three different scenes and outperforms the state-of-the-art methods.