Graph Neural Network (GNN) models are recently proposed to process the graph-structured data for the learning tasks on graphs, e.g., node classification, link prediction, and so on. This work focuses on the graph classification task, aiming to obtain the graph representation and predict the class label for a graph. Existing works proposed applying graph pooling to obtain graph embedding but still suffer from several issues. First, node embeddings are generated according to the topological information of the whole graph, but ignoring the local isomorphic substructures commonly seen in bioinformatics and chemistry. Another limitation arises when aggregating node embeddings. The hard assignment obtained through clustering algorithms, which rely on preset and fixed parameters instead of considering the graph's properties adaptively, restricts the flexibility in handling graphs of varying scales. To address the above problems, a module-based graph pooling framework (MGPool) is proposed in this work. Inspired by the rules of bioinformatics, MGPool assumes that a graph consists of multiple modules (also known as sub structures), which are identified based on the natural organization of the graph rather than the hard allocation of nodes. Benefiting from the hypothesis, MGPool generates node embeddings from graph-view and module-view, which is capable to capture global graph information and local isomorphic information respectively. Then module-level pooling is used to capture the intra-module information, while the inter-module information in terms of the correlation between modules is obtained through graph-level pooling. Finally, an entropy-based weighting mechanism is proposed to adjust the modules’ weights for the graph aggregation. Experiments conducted on bioinformatics benchmark datasets demonstrate the effectiveness of MGPool by outperforming other state-of-the-art graph pooling methods. For social network datasets, MGPool also provides competitive performance. Moreover, the visualization of module entropy weights is given to reveal the interpretability of the model.