Due to the poor functioning in daily living activities, community-dwelling older adults spend more time in their neighborhood environment. The perceived neighborhood environment is crucial to their quality of life (QoL). To explore the complex influences of perceived neighborhood environment on QoL, a questionnaire was designed to measure their perception of each factor of neighborhood environment and each domain of QoL. Based on collected data, the reliability test was applied to revise the questionnaire. Multiple linear regression (MLR) and structural equation modeling (SEM) were adopted to hypothesize and test the integrated model for community-dwelling older adults. The results show that community-dwelling older adults’ perceptions of neighbor support, facilities related to physical exercise and recreation, and accessibility to facilities impact their overall QoL with diverse coefficients of 0.437, 0.312, and 0.295, respectively; neighbor support (0.207) on physical health; sidewalk condition (0.134), natural environment (0.260), and facilities related to daily life (0.165) on psychological health; and neighbor support (0.298), facilities related to daily life (0.206), and design-related safety (0.225) on social relationship. This revealed that perceptions of neighborhood environment have diverse impacts on their QoL. This study can provide targeted retrofit strategies for communities to enhance QoL of community-dwelling older adults efficiently.