To precisely diagnose neurological diseases, such as Alzheimer's disease, clinicians need to observe the microstructural changes of local brain atrophy with the help of structural magnetic resonance image (sMRI). Some Convolutional Neural Networks (CNNs) have recently achieved excellent performance in auxiliary clinicians to provide the diagnosis suggestion. However, there still exist several challenges. Foremost, several researchers manually predefine some regions of interest (ROIs) as the input of the CNN-based networks, which impedes the model's robustness and interpretability of clinical applications. Second, since the position relevance of pathological features interferes with the surrounding tissue regions in ROIs, it is hard for the current CNN-based networks to extract the microstructural changes of these ROIs precisely. To address the above challenges, we optimize the Transformer structure for Alzheimer's Disease Diagnosis and propose an Inheritable Deformable Attention Network (IDA-Net). Specifically, the IDA-Net mainly comprises the 3D Deformable Self-Attention module and the Inheritable 3D Deformable Self-Attention module. The 3D Deformable Self-Attention module can automatically adjust the position and scale of the selected patches according to the structural changes in sMRI. Furthermore, the Inheritable 3D Deformable Self-Attention module can locate and output relatively important regions with discriminative features in sMRI, which can assist physicians in the clinical diagnosis. Our proposed IDA-Net method is evaluated on the sMRI of 2813 subjects from ADNI and AIBL datasets. The results show that our IDA-Net method behaves better than several state-of-the-art methods in classification performance and model generalization.