Thermal management is vital for the efficient and safe operation of alkaline electrolysis systems. Traditional alkaline electrolysis systems use simple proportional–integral–differentiation (PID) controllers to maintain the stack temperature near the rated value. However, in renewable-to-hydrogen scenarios, the stack temperature is disturbed by load fluctuations, and the temperature overshoot phenomenon occurs which can exceed the upper limit and harm the stack. This paper focuses on the thermal modeling and controller design of an alkaline electrolysis system under dynamic operating conditions. A control-oriented thermal model is established in the form of a third-order time-delay process, which is used for simulation and controller design. Based on this model, we propose two novel controllers to reduce temperature overshoot: one is a current feed-forward PID controller (PID-I), the other is a model predictive controller (MPC). Their performances are tested on a lab-scale system and the experimental results are satisfying: the temperature overshoot is reduced by 2.2 °C with the PID-I controller, and no obvious overshoot is observed with the MPC controller. Furthermore, the thermal dynamic performance of an MW-scale alkaline electrolysis system is analyzed by simulation, which shows that the temperature overshoot phenomenon is more general in large systems. The proposed method allows for higher temperature set points which can improve system efficiency by 1%.