Recently, generating visually secure cipher images by compressive sensing (CS) techniques has drawn much attention among researchers. However, most of these algorithms generate cipher images based on direct bit substitution and the underlying relationship between the hidden and modified data is not considered, which reduces the visual security of cipher images. In addition, performing CS on plain images directly is inefficient, and CS decryption quality is not high enough. Thus, we design a novel cryptosystem by introducing vector quantization (VQ) into CS-based encryption based on a 3D fractional Lorenz chaotic system. In our work, CS compresses only the sparser error matrix generated from the plain and VQ images in the secret generation phase, which improves CS compression performance and the quality of decrypted images. In addition, a smooth function is used in the embedding phase to find the underlying relationship and determine relatively suitable modifiable values for the carrier image. All the secret streams are produced by updating the initial values and control parameters from the fractional chaotic system, and then utilized in CS, diffusion, and embedding. Simulation results demonstrate the effectiveness of the proposed method.