Federated Learning coordinates multiple devices to train a shared model while preserving data privacy. Despite its potential benefit, the increasing number of participating devices poses new challenges to the deployment in real-world cases. The highly limited amount of data located on each device coupled with significantly unbalanced data across different devices severely impede the performance of the shared model and the overall training progress at the same time. In this paper, we propose FedHybrid, a hierarchical hybrid training framework for high-performance Federated Learning on a wide scale. Unlike the existing work that mainly focuses on the statistical challenge, FedHybrid establishes a hierarchical hybrid training framework that effectively utilizes the fragmented and unbalanced data located on the participating devices on a wide scale. Specifically, FedHybrid consists of the following two core components, a global coordinator deployed on the central server and a local coordinator deployed on each participating device. The global coordinator organizes the participating devices into different groups through jointly considering the system heterogeneity and unbalanced training data in order to accelerate the overall training progress while guaranteeing the model performance. Within each group, a novel device-to-device (D2D) sequential training procedure is coordinated by the local coordinator to effectively utilize the fragmented and unbalanced training data in order to intelligently update the local models. At the same time, we provide the theoretical analysis of FedHybrid and conduct extensive experiments to evaluate its effectiveness. The results show that FedHybrid effectively improves model accuracy up to 27% and accelerates the whole training process by 20% on average.