The convolutional neural network (CNN) is frequenctly used in silent speech recognition (SSR) based on surface electromyography (sEMG). Currently, there are two different modes when training the CNN classifier, using the sEMG signals from a single subject as the training datasets and using the mixed signals from multiple subjects as the training datasets. However, it still remains unclear how different training modes affect the performance of the CNN classifier in different classification metrics. In this study, two different training modes were used for the CNN classifier of SSR based on high-density sEMG (HD sEMG) signals. HD sEMG signals collected from six subjects were used to build two different training datasets. The HD sEMG signals from either a single subject or multiple subjects were to train the same CNN model and the performance difference was thoroughly compared in different metrics. The results showed that the CNN model trained from the signals of a single subject showed superior performance with higher average precision, average recall, and average F1 score. It also converged faster and was more stable under different signal conditions. However, it was only suitable for the SSR of the same subject, while the CNN model trained from the signals of multiple subjects showed satisfactiroy performance across all the recruited subjects. This study revealed that the CNN models trained with different training modes performed differently, and therefore the training mode could be taken into consideration in different applications of SSR based on sEMG.