Purpose: To investigate the changes in Chinese adults’ physical activity (PA) behavior and determinants before and during the COVID-19 pandemic. Method: A total of 1028 adults (aged 19–59 years) were recruited from 127 urban and rural neighborhoods in China using stratified three-stage probability sampling. Data collection was conducted in December 2019 and July 2020. Results: Compared with the data before the pandemic, individuals’ weekly moderate-to-vigorous-intensity PA (MVPA) decreased significantly from 139 min to 120 min, seven months after the outbreak (p = 0.01), with female and rural populations displaying a more significant decrease (p = 0.02). Overall, 13.7% of participants met the PA guidelines (World Health Organization) both before and during the pandemic, while 21.8% met the guidelines only before the pandemic and 18.1% increased their PA and met the PA guidelines during the pandemic. A total of 46.4% did not meet the PA guidelines before or during the pandemic. Determinants of PA behavior change before and during the pandemic included sports skills, self-determined motivation and support from sports organizations. Conclusions: The Chinese adults’ PA levels decreased significantly from before to during the COVID-19 pandemic, particularly among the female population. It is suggested that the enhancement of self-determined motivation, improvement of sport skills, and support from sports organizations might be effective in facilitating individuals’ engagement in PA during the pandemic.