Anelloviruses (AVs) are found in the vast majority of the human population and are most probably part of a healthy virome. These viruses infect humans in the early stage of life, however, the characteristics of the first colonizing AVs are still unknown. We screened a collection of 107 blood samples from children between 0.4 and 64.8 months of age for the presence of three AV genera: the Alpha-, Beta- and Gammatorquevirus. The youngest child that was positive for AV was 1.2 months old, and a peak in prevalence (100% of samples positive) was reached between the twelfth and eighteenth months of life. Intriguingly, the beta- and gammatorqueviruses were detected most at the early stage of life (up to 12 months), whereas alphatorqueviruses, the most common AVs in adults, increased in prevalence in children older than 12 months. To determine whether that order of colonization may be related to oral transmission and unequal presence of AV genera in breast milk, we examined 63 breast milk samples. Thirty-two percent of the breast milk samples were positive in a qPCR detecting beta- and gammatorqueviruses, while alphatorqueviruses were detected in 10% of the samples, and this difference was significant (p = 0.00654). In conclusion, we show that beta- and gammatorqueviruses colonize humans in the first months of life and that breastfeeding could play a role in AV transmission.