Automatic and accurate segmentation and classification of esophageal lesions are two essential tasks to assist endoscopists in Upper Gastrointestinal Endoscopy. However, there is no intelligent system that can diagnose more lesion types, handle multiple tasks simultaneously, and be more accurate in clinical work. Therefore, we present an innovative Multi-Task deep learning architecture named Attention Gating Pyramid Vision Transformer (AGPVT), which provides a solution for the accurate classification and precise segmentation of lesion types and regions simultaneously. The proposed AGPVT combines the benefits of cutting-edge deep learning model designs with Multi-Task Learning (MTL) in order to advance the field. Furthermore, a patch-wise multi-head attention gating method alongside a hybrid design MTL decoder, is employed as the core driving architecture of the AGPVT. Comprehensive experiments are conducted on a multicenter dataset which contains esophageal cancer, Barrett's esophagus, esophageal protruded lesions, esophagitis, and normal esophagus. Experimental results show that the proposed AGPVT achieves a classification accuracy of 96.84%, an IoU score of 85.61%, and a Dice score of 90.75%, outperforming existing methods and demonstrating its effectiveness in this domain.