Songs can express emotions and contain both lyrics and music. Early behavioral studies have shown that songs are influenced by the presence of lyrics for emotional processing. This work further examined the event-related potential (ERP) N400 responses of participants to elucidate whether lyrical and non-lyrical songs have different influences to different emotional states (happiness and sadness in this work). The song stimuli were processed and subjectively annotated with four conditions (i.e., lyrical and non-lyrical songs, and happy and sad emotional states), and the processed stimuli were presented to normal-hearing listeners in ERP experiments according to an affective priming paradigm. The song stimuli served as the prime stimuli and the affective-semantic words as the target stimuli. The experimental results showed that the non-lyrical songs evoked a larger amplitude than lyrical songs for happy emotion, and the lyrical songs evoked a larger amplitude than the non-lyrical songs for sad emotion. Results in this work suggest that the presence of the lyrics has a differential effect in happy and sad songs, and provide evidence that music and lyrics separately influence the diverse emotional processing.