Automatic modulation classification (AMC) based on deep learning (DL) is gaining increasing attention in dynamic spectrum access for 5G/6G wireless communications. However, inconsistent feature parameters between the training (source) and testing (target) data lead to performance degradation or even failure of existing DL-based AMC. The primary reason for this is the difficulty in obtaining sufficient labeled training data in the target domain. Therefore, we propose a novel cross-domain AMC algorithm based on multimodal information and transfer learning, utilizing abundant unlabeled target domain data. We achieve complementary gains by fusing multimodal information such as amplitude, phase, and spectrum, which are used to train a network. Additionally, we apply domain adversarial neural network technology from transfer learning to learn from a large number of unlabeled data samples in the target domain to address the issue of decreased accuracy in cross-domain AMC caused by differences in sampling rate, signal-to-noise ratio, and channel variations. Furthermore, we introduce class weight weighting and entropy weighting to solve the partial domain adaptation problem, considering that the target domain has fewer modulation signal classes than the source domain. Experimental results on two designed modulation datasets demonstrate improved performance gains, thus validating the effectiveness of the proposed method.