The task of classifying vowels via brain activities has been studied in many ideal direct-speech brain-computer interfaces (DS-BCIs). The vowels in those studies usually had clear acoustic differences, mainly on the first and second formants (i.e., F1 and F2). Whereas recent studies found that those speech features were difficult to be presented in DS-BCIs based on imagined speech, the spoken speech with audible output has the potential to provide insight regarding the relationship between spoken vowels' classification accuracies and their acoustic differences. This work aimed to classify four spoken Mandarin vowels (i.e., /a/, /u/, /i/ and /ü/, and pronounced with different consonants and tones to form monosyllabic stimuli in Mandarin Chinese) by using electroencephalogram (EEG) signals. The F1 and F2 of each spoken vowel were extracted; the corresponding spoken EEG signals were analyzed with the Riemannian manifold method and further used to classify spoken vowels with a linear discriminant classifier. The acoustic analysis showed that in the F1-F2 plane, the ll ellipse was closed to the /u/ and /i/ ellipses. The classification results showed that vowels /a/., /u/ and /i/ were well classified (82.0%, 69.5% and 68.2%, respectively), but vowel /ü/ was more easily classified into /u/, /i/ and /ü/. Results in this work suggested that the spoken vowels with similar formant structures were difficult to be classified by using their spoken EEG signals.