In this paper, performance of hybrid automatic repeat request with incremental redundancy (HARQ-IR) over Rayleigh fading channels is investigated. Different from prior analysis, time correlation in the channels is considered. Under time-correlated fading channels, the mutual information in multiple HARQ transmissions is correlated, making the analysis challenging. By using polynomial fitting technique, probability distribution function of the accumulated mutual information is derived. Three meaningful performance metrics including outage probability, average number of transmissions, and long term average throughput (LTAT) are then derived in closed-forms. Moreover, diversity order of HARQ-IR is also investigated. It is proved that full diversity can be achieved by HARQ-IR, i.e., the diversity order is equal to the number of transmissions, even under time-correlated fading channels. These analytical results are verified by simulations and enable the evaluation of the impact of various system parameters on the performance. Particularly, the results unveil the negative impact of time correlation on the outage and throughput performance. The results also show that although more transmissions would improve the outage performance, they may not be beneficial to the LTAT when time correlation is high. Optimal rate design to maximize the LTAT is finally discussed and significant LTAT improvement is demonstrated.