The crisis of exogenous contaminant residues in Chinese herbal medicines has attracted public attention. The conventional methods for contaminant detection require complex sample preparation and large-scale detection equipment. The development of simple detection methods is crucial for the quality of Chinese herbal medicines. However, most of the existing rapid detection methods can only detect a single target via one signal channel. But for many pollutants, it is far from enough. Therefore, we propose a novel detection method for the two pollutants by the dual-signal system. The detection system provides two signal channels for two contaminants’ quantification via fluorescence and visual detection. The established method displayed high selectivity even when the two targets co-exist. According to the recovery verification, the result shows that these two analytes were performed on complex substrates (ginseng radix et rhizoma, nelumbinis semen, and water). Smartphones are also being used to improve the accuracy of visual detection signals. On the smartphone platform, the determination relied on extracting the blue and green color values of the sample. In general, the designed two-step strategy provides a novel way to establish multi-scenario and multi-purpose rapid detection methods, which is conducive to the further development of contaminant detection in Chinese herbal medicines.