Drug discovery from complex mixture like Chinese herbs is a challenge and extensive false positives make the obtainment of specific bioactive compounds difficult. In the present study, a novel sample preparation method was proposed to rapidly reveal the specific bioactive compounds from complex mixtures using α-glucosidase as a case. Firstly, aqueous and methanol extracts of 500 traditional Chinese medicines were carried out with the aim of finding new sources of α-glucosidase inhibitors. As a result, the extracts of fruit of Terminalia chebula (FTC), flowers of Rosa rugosa (FRR) and Eugenia caryophyllata (FEC) as well as husk of Punica granatum (HPG) showed high inhibition on α-glucosidase. On-line liquid chromatography-diode array detection-tandem mass spectrometry and biochemical detection (HPLC-DAD-MS/MS-BCD) was performed to rapidly screen and characterize α-glucosidase inhibitors in these four extracts. After tentative identification, most of compounds with inhibitory activity in the investigated crude extracts were found to be tannins commonly recognized as non-specific enzyme inhibitors in vitro. Subsequently, the four extracts were treated with gelatin to improve specificity of the on-line system. Finally, two compounds with specific α-glucosidase inhibition were identified as corilagin and ellagic acid. The developed method could discover specific α-glucosidase inhibitors in complex mixtures such as plant extracts, which could also be used for discovery of specific inhibitors of other enzymes.