Direct speech brain-computer interface (DS-BCI) is an ideal way for speech communication by decoding signals collected from the brain. Electroencephalogram (EEG) has gained widespread use in DS-BCI studies due to its simplicity of operation and high temporal resolution. However, as human brain exhibits considerable inter-individual variability, classification models trained on the basis of data from one subject may not generalise well to other individuals, which is a major challenge in existing EEG signal classification studies. In this paper, the cross-subject classification performance of spoken Mandarin speech with EEG signals was investigated by using an end-to-end convolutional neural network (CNN) model pre-trained on the source data and fine-tuned on the target data. The raw EEG signals were directly used as the input to the model without using extracted features. In addition, adding Gaussian noise was used as the data augmentation method in order to deal with the unbalanced dataset. The proposed method was tested on a collected EEG dataset of spoken Mandarin speech, including vowel classification and tone classification tasks. The average classification accuracies of four vowels and four tones were 63.1% and 51.7% respectively. The average accuracy of tone classification was significantly improved compared with the machine learning and subject-dependent methods. The results of this work showed the potential of the fine-tuning based CNN model in the cross-subject studies of EEG decoding.