The built environment is among the critical factors in older adults’ travel behavior, and a favorable built environment can encourage them to travel and engage in various activities. Existing studies have mostly focused on exploring the correlation between the built environment and travel behavior, ignoring the heterogeneity between the two at different times of the day. In this study, we conducted structured, face-to-face interviews in the Nanjing (China) metropolitan area to investigate the time consumed per trip by older adults using various travel modes and used the structural equation and random forest models to explore the relationship between the built environment and older adults’ travel time. The results demonstrated that older adults had different perspectives on travel during different time periods. Different environments and the convenience of destinations affected their overall satisfaction during travel. We found a nonlinear relationship between the built environment and travel time. Metropolitan street connectivity initially had a positive effect on travel time until a certain threshold or peak, whereafter a gradual decline ensued. This nonlinear relationship also existed between the proportion of green space and the distance to subway stations. These results can guide the retrofitting and construction of age-friendly metropolitan infrastructure facilities that promote older adults’ mobility.