With the increasing use of media in communications, the content security of digital images attracts much attention in both the academia and industry. Meanwhile, for the symmetric cryptosystem, the key transmission and management is burden on users. This paper proposes an asymmetric image encryption algorithm based on an elliptic curve cryptosystem (ECC). The sender and recipient agree on an elliptic curve point based on Diffie-Hellman public key sharing technique. Firstly, to reduce the encryption times, the sender groups pixel values together, and converts them into big integers; secondly, the sender encrypts big integers with ECC and the chaotic system; finally, the encrypted image is obtained from encrypted big integers. The proposed algorithm makes the key transmission and management relatively simple and secure. Simulation data show that the proposed algorithm exhibits both the strong security and high efficiency.