Modern distribution grids are currently being challenged by frequent and sizable voltage fluctuations, due mainly to the increasing deployment of renewable generation. Considering equipment characteristics, traditional devices (e.g., on-load tap changers) to maintain bus voltages cannot provide frequent regulations. To deal with short-term voltage fluctuations, this paper proposes to cooperatively control reactive and active power through PV inverters and district cooling systems. However, traditional voltage control optimization relies heavily on accurate physical models (e.g., network topology), and brings huge computation burdens with enlarging system scale. In this context, this paper adopts model-free reinforcement learning (RL) to solve the controller without prior knowledge of system models. However, because voltage violations occur not frequently in practice, the irregular occurrence brings sparse reward and biased-distribution experience issues in RL training. Hence, on top of the traditional actor–critic structure, we propose two improvements: 1) a compensator module is designed to cope with the sparse reward issue; 2) a scenario-classified experience replay method is proposed for RL training sampling, which can correct the experience distribution to improve training efficiency for a typical scenario with violated voltages. Numerical studies on a 33-bus network show that, the proposed method can smooth voltage fluctuations better, with negligible temperature impacts on demand-side users.