Modality differences and intra-class differences have been hot research problems in the field of cross-modality person re-identification currently. In this paper, we propose a cross-modality person re-identification method based on joint middle modality and representation learning. To reduce the modality differences, a middle modal generator is used to map different modal images to a unified feature space to generate middle modality images. A two-stream network with parameter sharing is used to extract the combined features of the original image and the middle modality image. In addition, a multi-granularity pooling strategy combining global features and local features is used to improve the representation learning capability of the model and further reduce the modality differences. To reduce the intra-class differences, the model is further optimized by combining distribution consistency loss, label smoothing cross-entropy loss, and hetero-center triplet loss to reduce the intra-class distance and accelerate the model convergence. In this paper, we use the publicly available datasets RegDB and SYSU-MM01 for validation. The results show that the proposed approach in this paper reaches 68.11% mAP in All Search mode for the SYSU-MM01 dataset and 86.54% mAP in VtI mode for the RegDB dataset, with a performance improvement of 3.29% and 3.29%, respectively, which demonstrate the effectiveness of the proposed method.