Due to the increasing fatal traffic accidents, there are strong desire for more effective and convenient techniques for driving fatigue detection. Here, we propose a unified framework E-Key to simultaneously perform personal identification (PI) and driving fatigue detection using a convolutional attention neural network (CNN-Attention). The performance was assessed using EEG data collected through a wearable dry-sensor system from 31 healthy subjects undergoing a 90-min simulated driving task. In comparison with three widely-used competitive models (including CNN, CNN-LSTM, and Attention), the proposed scheme achieved the best (p < 0.01) performance in both PI (98.5%) and fatigue detection (97.8%). Besides, the spatial-temporal structure of the proposed framework exhibits an optimal balance between classification performance and computational efficiency. Additional validation analyses were conducted to assess the reliability and practicability of the model via re-configuring the kernel size and manipulating the input data, showing that it can achieve a satisfactory performance using a subset of the input data. In sum, these findings would pave the way for further practical implementation of in-vehicle expert system, showing great potential in autonomous driving and car-sharing where currently monitoring of PI and driving fatigue are of particular interest.