Automatic segmentation of the cancerous esophagus in computed tomography (CT) images is a computer-assisted method that can improve the efficiency of the diagnosis and treatment. Due to the diversity of the cancer stage and location, the anatomical structure of the cancerous esophagus is various. Moreover, the low contrast against surrounding tissues leads to a blurry boundary of the cancerous esophagus. Therefore, existing segmentation networks cannot achieve satisfactory results in automatic segmentation of the cancerous esophagus. In this article, we propose a novel 2.5D segmentation network named Eso-Net for the cancerous esophagus based on an encoder-decoder architecture. A 3D enhancement filter called Multi-Structure Response Filter (MSRF) is designed to extract 3D structural information as prior knowledge. Furthermore, dilated convolutions and residual connections are employed in the convolutional blocks of Eso-Net for multi-scale feature learning. With 3D structural priors, Prior Attention Modules (PAM) are incorporated into the network to facilitate the transmission of relevant spatial information. The experiments are conducted on the dataset from 30 esophageal cancer patients, and we report an 84.839% dice similarity coefficient, an 85.955% precision, an 83.752% sensitivity, and a 2.583mm Hausdorff distance. The experimental results demonstrate that the proposed method outperforms other existing segmentation networks in this task and can effectively assist doctors in the diagnosis and treatment of esophageal cancer.