Most of the existing research for cross-domain fault diagnosis of rotating machineries focuses on the single-source-single-target domain adaptation. The single-source-single-target domain adaptation refers to applying the model and knowledge learned from the source domain from a certain working condition to the target domain from another working condition, which is different from the source domain. In addition, only one working condition is considered in the source domain and target domain of a fault diagnosis task. Compared with the fault diagnosis task of single-source-single-target domain adaptation, the multi-source-multi-target domain adaptation is more complicated and insufficiently researched. The differences between the source and target domains have to be considered, but they are seldom modelled in the existing methods. To solve the multi-source-multi-target fault diagnosis problem, this paper proposes a two-stage training multi-branch network (TTMN). In the source-only learning stage of the TTMN, a one-dimensional convolutional neural network (1D-CNN) is trained by all the source domain data, and the similarity of each one-source-one-target pair is measured. In the similarity-weighted domain adaptation stage of the TTMN, each target domain is assigned to one branch transferred from the well-trained 1D-CNN in the source-only learning stage and fine-tuned by the cosine similarity-weighted loss. Thus, the fault diagnosis with multi-source-multi-target domain adaptation is transformed into a single-source-single-target domain diagnosis. The proposed TTMN is validated on the Donghua University (DHU) and the Case Western Reserve University (CWRU) bearing datasets. The results show the effectiveness and outperformance of the proposed method compared with some state-of-the-art methods.