Segmentation of roads in remote sensing (RS) images is a challenging task due to the inhomogeneous intensity, nonconsistent contrast, and very cluttered background in remote sensing images. Recent approaches, mostly relying on convolutions or self-attention, make it difficult to extract weak and continuous road objects. Fourier neural operators (FNOs) provide another novel mechanism for capturing long-range and fine-grained features beyond self-attention. Based on it, we propose an adaptive Fourier convolution network (AFCNet) on the spatial-spectral domain for road segmentation in this article. The AFCNet is built on the pipeline of the classical U-Net model and its core is the proposed Fourier neural encoder (FNE), which is built on a feed-forward layer and a flexible Fourier convolutional structure composed of Fourier-domain pooling layers, asymmetric convolutions, squeeze-excitation inspired self-attention, and adaptive multiscale fusion (AMF) layers. Furthermore, we combine the FNE and bottleneck in ResNet to form a hybrid global-local feature representation scheme to capture the long and weak road objects in remote sensing images. The experiments on two public datasets, the Massachusetts Roads and DeepGlobe Road datasets, have shown that AFCNet worked with fewer parameters and outperformed most previous methods in terms of accuracy, precision, recall, and mean intersection over union (mIoU).