The COVID-19 pandemic negatively influenced individuals’ physical activity levels (PALs) and particularly the PAL of the elderly. However, few studies have examined the correlates of PALs in this population during the pandemic. This study aimed to evaluate the residence-specific correlates of PALs in elderly people from Croatia and Bosnia and Herzegovina during the COVID-19 pandemic. The participants were 211 persons older than 65 years (101 females), of whom 111 were community-dwelling residents, and 110 were nursing home residents (71.11 ± 3.11 and 72.22 ± 4.01 years of age, respectively; t-test = 0.91, p < 0.05). The variables included health status, residential status sociodemographic factors, anthropometrics (body mass, height, and body mass index), and PAL. PAL was evaluated using a translated version of the Physical Activity Scale for the Elderly (PASE), and was validated in this study. PASE showed good test–retest reliability (51% of the common variance) and validity (57% of the common variance, with the step count measured using pedometers). Apart from participants’ health status and age, PAL was positively correlated with (i) community-dwelling residence (OR = 1.93, 95% CI: 1.60–2.23), and (ii) a lower BMI (OR = 0.85, 95% CI = 0.71–0.98). The pre-pandemic physical activity was positively correlated with the PAL of the nursing home residents (OR = 1.2, 95% CI: 1.02–1.45). A higher education level was positively correlated with the PAL of community-dwelling residents (OR = 1.31, 95% CI: 1.04–1.66). This study evidenced the residence-specific correlates of PALs, and enabled the identification of specific groups that are at risk of having low PALs during the pandemic. Future studies examining this problem during a non-pandemic period are warranted.