Detecting increases of incidence rate of an adverse health event is critical to health surveillance. Motivated by successful applications in industry quality control and productivity improvement, various cumulative sum (CUSUM) procedures have been developed to monitor rate changes of health events when population size changes over time. Average run length, which is commonly used in quality control, may be inappropriate in some situations when comparing control chart performance in health surveillance applications. This paper provides a thorough statistical comparison of CUSUM‐type procedures in the health care context based on different evaluation metrics. We show that different measures may favor different surveillance methods depending on the corresponding design criteria. We thus recommend a probabilistic measure as the criterion for design and evaluation of CUSUM control charts in health care surveillance.