We consider the problem of artifacts in electroencephalography (EEG) data. In a practical motor imagery based brain-computer interface (BCI) system, EEG signals are usually contaminated by misleading trials caused by artifacts, measurement inaccuracies, or improper imagination of a movement. As a result, the performance of a BCI system can be degraded. In this paper, we introduce a novel algorithm combining Gaussian mixture model (GMM) and genetic algorithm (GA) to detect the abnormal EEG samples. In addition, this algorithm can be also integrated with other data-driven feature exaction method (e.g., common spatial pattern (CSP)) so that a more reliable analysis can be obtained by pruning the potential outliers and noisy samples, and consequently the performance of a BCI system can be improved. Experimental results demonstrate significant improvement in comparison with the conventional mixture model. © 2011 Elsevier Ltd. All rights reserved.