In recent decades, there has been considerable popularity in employing nonlinear dynamics and permutation-substitution structures for image encryption. Three procedures generally exist in such image encryption schemes: the key schedule module for producing encryption elements, permutation for image scrambling and substitution for pixel modification. This paper cryptanalyzes a family of image encryption schemes that adopt pixel-level permutation and modular addition-based substitution. The security analysis first reveals a common defect in the studied image encryption schemes. Specifically, the mapping from the differentials of the ciphertexts to those of the plaintexts is found to be linear and independent of the key schedules, permutation techniques and encryption rounds. On this theory basis, a universal chosen-ciphertext attack is further proposed. Experimental results demonstrate that the proposed attack can recover the plaintexts of the studied image encryption schemes without a security key or any encryption elements. Related cryptographic discussions are also given.