With the development of commercial buildings in modern cities, the district cooling system (DCS) is rapidly increasing due to its high efficiency for providing cooling services to multiple buildings. By utilizing buildings' inherent thermal inertia, DCS has huge potential to participate in the electricity market and provide regulation capacity. However, offering the DCS's regulation capacity ahead of the operating hour is quite challenging. Its available capacity is changing with time significantly due to multiple commercial buildings' stochastic cooling demand and the electricity market's uncertain signals. To address this issue, this paper proposes a strategy framework to offer the DCS's available regulation capacity for achieving the maximum revenue while respecting the users' comfortable indoor temperature requirements. First, the DCS's revenue model is developed based on its regulation capacity and performance score in the electricity market. Then, the regulation capacity offering strategy in each time slot is formulated as a Markov Decision Process (MDP). On this basis, the deep determined policy gradient algorithm is implemented to iterate the policy in the MDP to obtain the optimal results, which requires no knowledge of the uncertainties or physical model. Finally, we use the realistic RegA frequency regulation signals from PJM market to validate that the proposed strategy is effective in evaluating the system's available capacity with high-quality performance.