Reconfigurable intelligent surface (RIS) has been envisioned as an innovative technology to assist millimeter wave (mmWave) communications. Thanks to both advantages of low hardware cost and low power consumption, the hybrid transceiver structure also becomes an integral component of mmWave systems. However, due to practical limitations of hardware components, the RIS-assisted mmWave communications usually suffer unavoidable hardware impairments (HWIs). In this paper, we aim to minimize the (sum) MSE and maximize the average rate of the hardware-impaired RIS-assisted point-to-point mmWave MIMO system, respectively, by jointly optimizing the hybrid transceiver and RIS reflection coefficients under the realistic discrete phase shift constraints. We firstly consider the single-antenna user case and propose efficient alternating optimization (AO) algorithms to solve the two intractable problems. A binary-oriented exact penalty (BEP) method is developed for the involved discrete optimization, which is able to strike a good trade-off between performance and complexity. Moreover, we analyze the optimality of AO algorithms under the cascaded line-of-sight (LoS) channel condition, and reveal both the MSE floor effect and average rate saturation effect in the high-SNR regime. The above studies are then extended to the general multi-antenna user case, where a low-complexity two-phase scheme with the aim of creating the favorable RIS-cascaded channel in the first phase and enhancing system performance in the second phase is proposed. This two-phase scheme is also demonstrated to attain the optimal performance in the LoS scenario. Numerical results validate our theoretical analysis and illustrate superior performance of the proposed algorithms over various benchmark schemes.