This paper presents a new encryption scheme of protecting medical images. It has high efficiency and shows robustness of defending some impulse noise and data loss. First, some random data are inserted into surroundings of the image. Then, two rounds of high-speed scrambling and pixel adaptive diffusion are performed to randomly shuffle neighboring pixels and spread these inserted random data over the entire image. The proposed encryption scheme can be directly applied to medical images with any representation format. We provide two kinds of operations to implement the pixel adaptive diffusion: bitwise XOR and modulo arithmetic. The former has high efficiency in hardware platforms while the latter can achieve fast speed in software platforms. Simulations and evaluations show that both encryption schemes using bitwise XOR and modulo arithmetic have high security levels, can achieve much faster speeds, and can better adapt to impulse noise and data loss interference than several typical and state-of-the-art encryption schemes. (C) 2017 Elsevier B.V. All rights reserved.