Aging in place enables seniors to live at home and in their familiar neighborhoods as long as their health conditions allow. To date, few efforts have been made to examine the linkage between the physical and social aspects of the neighborhood environment, the activity-travel patterns, and their overall effects on life satisfaction. Therefore, this study aims to contribute to the existing literature by examining the relationship between older people’s person–environment fit, activity-travel patterns, sense of community, and life satisfaction. Seemingly unrelated regression was applied to analyze questionnaire survey data collected from six community centers for older people in Hong Kong. Using the geographical perspective, the sense of community is further divided into the people and place aspects, with the latter found to have a greater effect on overall life satisfaction. Subjective walkability both directly and indirectly influences life satisfaction. Satisfaction with public transport facilities, health care facilities, and greenery, parks, and promenades indirectly influences older people’s life satisfaction through their sense of community. The percentage of out-of-home activity time spent in the common neighborhood has a strong positive relationship with the sense of community. The findings highlight the importance of high-quality pedestrian infrastructure and public transport facilities in planning for aging in place. After identifying the common neighborhoods, local governments can listen to older people’s needs and make improvements on the infrastructure, services, and facilities in common neighborhoods, as well as the connectivity to these areas.