Background: Umbilical venous catheters (UVCs) are the standard of care in neonatal intensive care units (NICUs) to administer fluids, parenteral nutrition and medications, although complications may occur, including central line-associated blood stream infections (CLABSIs). However, the dwell time to reduce CLABSI risk remains an open issue. Methods: We performed a single-center retrospective study of newborns hospitalized in the Modena NICU with at least one UVC inserted over a 6-year period (period 1: January 2011–December 2013; period 2: January 2019–December 2021). We selected a non-consecutive 6-year period to emphasize the differences in UVC management practices that have occurred over time in our NICU. The UVC dwell time and catheter-related complications during the first 4 weeks of life were examined. Results: The UVC dwell time was shorter in period 2 (median 4 days vs. 5 days, p < 0.00001). Between the two periods, the incidence of CLABSIs remained unchanged (p = 0.5425). However, in period 2, there was an increased need for peripherally inserted central catheters (PICCs) after UVC removal, with a rise in PICC infections after UVC removal (p = 0.0239). Conclusions: In our NICU, shortening UVC dwell time from 5 to 4 days did not decrease the UVC-related complications. Instead, the earlier removal of UVCs led to a higher number of PICCs inserted, possibly increasing the overall infectious risk.