Urban traffic congestion is not only an important cause of traffic accidents, but also a major hinder to urban development. By learning the historical traffic flow data, we can forecast the traffic flow of some regions in the future, which is of great significance to urban road planning, traffic management, traffic control and many more. However, due to the complex topology of traffic network and the diversity of influencing factors to traffic flow, the traffic modes are usually complicated and volatile, which makes traffic flow prediction very difficult. In this paper, we propose a new graph convolution neural network, namely Multi-mode Dynamic Residual Graph Convolution Network (MDRGCN), to capture the dynamic impact of different factors on traffic flow in a road network simultaneously. Firstly, we design a multi-mode dynamic graph convolution module (MDGCN), which is employed to capture the impact of different traffic modes by learning two types of relationship matrices. Then, we design a multi-mode dynamic graph convolution gated recurrent unit (MDGRU) to realize the combination of spatial and temporal dependences. Finally, we use a dynamic residual module (DRM) to integrate the orginal traffic data and the spatio-temporal features extracted by the MDGRU module to forecast the future traffic flow. Experimental reulsts conducted on the NYCTaxi and NYCBike datasets validate that the MDRGCN model performs better than the other eight baselines.