In this paper, the security of a cryptosystem based on phase truncation and a designed amplitude modulator (AM) is evaluated. In the cryptosystem, an undercover AM used as an additional key is added to modulate the amplitude information of the spectrum in the Fourier plane. Compared to the conventional phase-truncated Fourier transform (PTFT)-based cryptosystem, the security of the cryptosystem is improved by increasing the number of unknown keys. However, it is found that the designed AM is irrelative to the plaintext, and one of the parameters in the designed AM contributes less to the security enhancement of the cryptosystem due to low key sensitivity. Based on the analysis, a special attack containing two iterative processes is proposed to crack the cryptosystem, in which the known-plaintext-attack-based iterative process I with a specific normalization operator is used to retrieve the designed AM and the amplitude-phase-retrieval-technique-based iterative process II is used to retrieve the corresponding plaintext from the arbitrarily given ciphertext with the help of the retrieved AM. In addition, an inherent drawback widely existing in PTFT-based cryptosystems is reported for the first time: most information of the original image could be retrieved using two correct phase keys (or only the first phase key) generated in the encryption process, even without the corresponding ciphertext in PTFT-based cryptosystems. To address this issue, a security-enhanced cryptosystem is proposed in this paper. Numerical simulation is carried out to demonstrate the effectiveness and feasibility of the proposed attack and cryptosystem.