Urban informal settlements (UISs) are high-density population settlements with low standards of living and supply. UIS semantic segmentation, which identifies pixels corresponding to informal settlements in remote sensing images, is crucial to the estimation of poor communities, urban management, resource allocation, and future planning, particularly in megacities. However, most studies on informal settlement mapping are either based on parcels (image classification) or pixels (semantic segmentation). Few studies utilize object information to improve UIS mapping. Since informal settlements are formed by buildings (objects), utilizing object information can improve UIS semantic segmentation. Furthermore, current UIS mapping studies mainly focus on using single-modality remote sensing images, and there is a lack of related research on using multimodal data. Due to the spatial heterogeneity of informal settlements, using only a single modality of remote sensing image features limits the effectiveness and accuracy of informal settlements semantic segmentation. Aiming at achieving fine-scale UIS mapping results, this article proposes a UIS semantic segmentation method, namely UisNet, that utilizes a transformer-based block to receive multimodal data, including high-spatial-resolution remote sensing images (parcel- and pixel-level) and building polygon data (object-level) to identify UIS. The experiments were conducted in Shenzhen City, and they confirmed the superior performance of UisNet, which achieved an overall accuracy (OA) of 94.80% and a mean intersection over union (mIoU) of 85.51% in the testing set of the manually labeled UIS semantic segmentation dataset (UIS-Shenzhen dataset) and outperformed the best models on semantic segmentation tasks. Besides, we add a set of experiments on a public dataset [gaofen image dataset (GID) dataset] and compare our method with the current state-of-the-art semantic segmentation methods. Experiments show that the proposed UisNet improves mIoU by 1.64% to 7.58% compared to other methods. This work will be available at https://github.com/RunyuFan/.