The Graph Convolutional Network (GCN) has garnered substantial interest over an extended period owing to its notable efficacy in addressing topological correlations, with particular achievements observed in skeleton-based action recognition. However, it is intriguing to note that the efficacy of the adaptive module within the existing model gradually diminishes as the learning capacity of the adjacency matrix improves. Inspired by this, AL-GCN constructs a comprehensively learnable group adjacency matrix in both spatial and temporal dimensions, thus forming an elegant and efficient GCN-based model. Firstly, the prevalent adaptive module is courageously removed, and the key points of the human body are astutely leveraged as motion compensation to provide dynamic correlation support. Secondly, a similar temporal adjacency matrix group is designed in the temporal domain to capture the global interaction characteristics more effectively. Subsequently, attention modules are purposefully crafted for both the temporal and spatial domains, which provide discriminative information between classes and improve learnability and interpretability. Finally, the Bayesian weight selection algorithm is utilized to achieve efficient and accurate fusion results for multi-stream data. On the NTU-60, NTU-120, and NW-UCLA, AL-GCN outperforms the state-of-the-art method, with up to 7.0% improvement on the challenging UAV-Human, achieving superior performance in multiple settings, all while maintaining a lower computational cost. Related code will be available on Auto-Learning-GCN.