Shadow removal is a challenging computer vision and multimedia task that aims to restore image content in shadow regions. The state-of-the-art shadow removal methods introduce artifacts near shadow boundaries or inconsistencies between shadow and nonshadow areas, which can be easily noticed by the human eye at first glance. In this paper, we design a boundary-aware shadow removal network (BA-ShadowNet) that improves shadow removal accuracy by increasing the removal performance at shadow boundaries. In contrast with previously developed methods, which usually consider shadow boundary optimization to be a postprocessing technique, our method performs shadow removal and shadow boundary optimization simultaneously. For this purpose, the proposed BA-ShadowNet is designed as a multiscale encoder-decoder structure, where the decoder consists of a shadow removal branch and a shadow optimization branch. An interaction module is then introduced to fuse and exchange the features of the two branches. This module facilitates the removal branch in perceiving the locations and colors of shadow boundaries. Additionally, it optimizes the boundary branch according to the image context extracted from the removal branch. A three-term loss function is further developed to supervise the shadow removal results and to address the issue of imbalanced supervision between shadow boundary pixels and pixels inside shadows. Extensive experiments conducted on the ISTD+ and SRD datasets demonstrate that the proposed BA-ShadowNet greatly outperforms the state-of-the-art methods with respect to shadow removal.