Background: Antibiotics and antibiotic resistance genes (ARGs) used in intensive animal farming threaten human health worldwide; however, the common resistome, ARG mobility, and ARG host composition in different animal manures and mixed manure composts remain unclear. In the present study, metagenomic assembly and cross-sample mapping were used to comprehensively decipher the common resistome and its potential mobility and hosts in animal manure and composts. Results: In total, 201 ARGs were shared among different animal (layer, broiler, swine, beef cow, and dairy cow) manures and accounted for 86–99% of total relative abundance of ARGs. Except for multidrug, sulfonamide, and trimethoprim resistance genes, the relative abundance of most ARGs in composts was significantly lower than that in animal manure. Procrustes analysis indicated that antibiotic residues positively correlated with ARG composition in manure but not in composts. More than 75% ARG subtypes were shared between plasmids and chromosomes in our samples. Transposases could play a pivotal role in mediating the transfer of ARGs between different phyla in animal manure and composting. Cross-sample mapping to contigs carrying ARGs showed that the hosts of common resistome in manure had preference on animal species, and the dominant genus of ARG host shifted from Enterococcus in manure to Pseudomonas in composts. The broad host range and linking with diverse mobile genetic elements (MGEs) were two key factors for ARGs, such as sul1 and aadA, which could survive during composting. The multidrug resistance genes represented the dominant ARGs in pathogenic antibiotic-resistant bacteria in manure but could be effectively controlled by composting. Conclusions: Our experiments revealed the common resistome in animal manure, classified and relative quantified the ARG hosts, and assessed the mobility of ARGs. Composting can mitigate ARGs in animal manure by altering the bacterial hosts; however, persistent ARGs can escape from the removal because of diverse host range and MGEs. Our findings provide an overall background for source tracking, risk assessment, and control of livestock ARGs.