This study creatively introduces a class of adaptive Fourier decomposition (AFD) techniques into the field of image encryption, where AFD and its variants are newly developed signal decomposition methods. The novelty of this article is twofold. First, we use three different AFD techniques, including Core AFD, Unwinding AFD, and Cyclic AFD, to generate two key streams through decomposing a source image. The two key streams generated by different AFD algorithms are different and aperiodic, which can effectively improve the security of encryption algorithm. Furthermore, this work also proposes a novel encryption method that implements the confusion and diffusion operations by index-sorted mapping method and pixel swapping operation based on two generated key streams, respectively. To ensure that the image is fully scrambled, both pixel-level and bit-level permutations are performed, which employs a novel bilateral sequence diffusion method to change the image pixel distribution based on pixel swap operations. The cryptographic performance of the scheme is evaluated through various security analyses, including key sensitivity, statistical, entropy, clipping attack, and differential attack analysis. Experimental results confirm that the proposed image encryption scheme ensures a high level of security and exhibits superior performance in resisting various attacks compared with several traditional and state-of-the-art image encryption methods.