Numerous viruses hijack cellular protein trafficking pathways to mediate cell entry or to rearrange membrane structures thereby promoting viral replication and antagonizing the immune response. Adaptor protein complexes (AP), which mediate protein sorting in endocytic and secretory transport pathways, are one of the conserved viral targets with many viruses possessing AP-interacting motifs. We present here different mechanisms of viral interference with AP complexes and the functional consequences that allow for efficient viral propagation and evasion of host immune defense. The ubiquity of this phenomenon is evidenced by the fact that there are representatives for AP interference in all major viral families, covered in this review. The best described examples are interactions of human immunodeficiency virus and human herpesviruses with AP complexes. Several other viruses, like Ebola, Nipah, and SARS-CoV-2, are pointed out as high priority disease-causative agents supporting the need for deeper understanding of virus-AP interplay which can be exploited in the design of novel antiviral therapies.