As a naturalistic form of communication, direct-speech brain-computer interfaces (DS-BCIs) give users the possibility of 'reading the mind'. The understanding of brain processing in the spoken speech is the bridge to the ideal DS-BCI, and lexical tones as an important element in tone languages like Mandarin are desirable to be well explored. This work studied the classification of four Mandarin tones in spoken speech by using electroencephalogram (EEG). Specially, a speech pronunciation experiment was performed to include imagined, intended, and spoken states. The multiple combinations of vowels, consonants, and tones constituted monosyllables as the stimuli. Common spatial pattern (CSP) and Riemannian manifold were used as feature extraction methods, and linear discriminant analysis as classifier. Result showed that the four-class classification accuracy of the Riemannian manifold-based method across all participants was 42.9%, which was 12.3% higher than that of the CSP-based method. This work suggested that the spoken Mandarin tones were decodable with corresponding EEG signals.