Performing joint genomic predictions for multiple breeds (MBGP) to expand the reference size is a promising strategy for improving the prediction for limited population sizes or phenotypic records for a single breed. This study proposes an MBGP model—mbBayesAB, which treats the same traits of different breeds as potentially genetically related but different, and divides chromosomes into independent blocks to fit heterogeneous genetic (co)variances. Best practices of random effect (co)variance matrix priors in mbBayesAB were analyzed, and the prediction accuracies of mbBayesAB were compared with within-breed (WBGP) and other commonly used MBGP models. The results showed that assigning an inverse Wishart prior to the random effect and obtaining information on the scale of the inverse Wishart prior from the phenotype enabled mbBayesAB to achieve the highest accuracy. When combining two cattle breeds (Limousin and Angus) in reference, mbBayesAB achieved higher accuracy than the WBGP model for two weight traits. For the marbling score trait in pigs, MBGP of the Yorkshire and Landrace breeds led to a 6.27% increase in accuracy for Yorkshire validation using mbBayesAB compared to that using the WBGP model. Therefore, considering heterogeneous genetic (co)variance in MBGP is advantageous. However, determining appropriate priors for (co)variance and hyperparameters is crucial for MBGP.