Caffeine is commonly used to excess by the general public, and most pregnant women drink caffeine on a daily basis, which can become a habit. Maternal caffeine intake during pregnancy is associated with severe gestational outcomes. Due to its lipophilic nature, caffeine can cross the blood–brain barrier, placental barrier, and even amniotic fluid. It can be found in substantive amounts in breast milk and semen. There has been a reported drop in neonatal anthropometric measurements with increased caffeine consumption in some cohort studies. This narrative review using literature titles and abstracts from the electronic databases of PubMed, Embase, and Scopus investigates the data linking maternal caffeine use to unfavorable pregnancy outcomes. It also evaluates the validity of the recommendations made by health professionals on caffeine consumption by mothers from the available literature. The results of our comprehensive literature search of case–control studies, cohort studies, randomized control trials, and meta-analyses, imply that caffeine use during pregnancy is linked to miscarriage, stillbirth, low birth weight, and babies that are small for gestational age. It was also found that there may be effects on the neurodevelopment of the child and links to obesity and acute leukemia. These effects can even be seen at doses well below the daily advised limit of 200 mg. The genetic variations in caffeine metabolism and epigenetic changes may play a role in the differential response to caffeine doses. It is crucial that women obtain solid, evidence-based guidance regarding the possible risks associated with caffeine.