Automatic modulation classification (AMC) is an essential part in a cognitive radio receiver. Benefited from the discriminative constellation characteristics among most modulations, AMC methods based on constellation diagrams usually achieve pleasant performance. However, in noncooperation communication systems, constellation diagrams expressing modulations explicitly are difficult to obtain via blind symbol timing synchronization, especially in complicated wireless channels. Therefore, this article proposes a novel constellation diagram-based AMC architecture called attentive Siamese networks (ASNs) by considering multitiming constellation diagrams (MCDs) and selecting the proper symbol timings at the feature level, which is a more robust way than the conventional signal-level symbol timing synchronization. In detail, convolutional neural networks sharing the same parameters first extract deep feature vectors for MCDs. Then, an attention inference module weights all the deep feature vectors. Finally, AMC is realized based on the weighted feature vectors. Moreover, the ASN architecture can be trained end-to-end. Comparing with the state-of-the-art methods that take diverse representations of received baseband signals as input, experimental results based on the RadioML 2018.01A dataset and non-Gaussian noise dataset demonstrate that ASN achieves a remarkable improvement, whose classification accuracy goes over 99% when the signal-to-noise ratio (SNR) > 10 dB.