Protein–protein interactions (PPIs) between modular binding domains and their target peptide motifs are thought to largely depend on the intrinsic binding specificities of the domains. The large family of SRC Homology 3 (SH3) domains contribute to cellular processes via their ability to support such PPIs. While the intrinsic binding specificities of SH3 domains have been studied in vitro, whether each domain is necessary and sufficient to define PPI specificity in vivo is largely unknown. Here, by combining deletion, mutation, swapping and shuffling of SH3 domains and measurements of their impact on protein interactions in yeast, we find that most SH3s do not dictate PPI specificity independently from their host protein in vivo. We show that the identity of the host protein and the position of the SH3 domains within their host are critical for PPI specificity, for cellular functions and for key biophysical processes such as phase separation. Our work demonstrates the importance of the interplay between a modular PPI domain such as SH3 and its host protein in establishing specificity to wire PPI networks. These findings will aid understanding how protein networks are rewired during evolution and in the context of mutation-driven diseases such as cancer.