In recent decades, the introduction of chaos to image encryption has drawn worldwide attention. The permutation-substitution architecture has been widely applied, and chaotic systems are generally employed to produce the required encryption elements. Although many security assessment tests have been conducted, some chaotic image ciphers are being cryptanalyzed. In this article, we evaluate the security of a family of image ciphers whose encryption kernel consists of a bit-level or pixel-level permutation and a bit-wise exclusive OR substitution. After investigating the intrinsic linearity inside the outfitted structures and encryption techniques, we find that each ciphertext-plaintext pair can be represented as a combination of a set of ciphertext-plaintext bases. A chosen-ciphertext attack is proposed to construct the ciphertext-plaintext bases rather than the traditional solution to retrieve equivalent encryption elements. We further reveal that such weakness cannot be remedied by common enhancements such as more chaotic dynamics, complex permutation methods, and random pixel insertion during encryption. In addition, applications of the proposed attack to break 12 ciphers are theoretically presented and experimentally verified.