The COVID-19 pandemic has caused an abrupt change in lifestyle for many people with restrictions, often leading to a decrease in physical activity (PA), and thus contributing to a negative perception of health status. The purpose of this study was to examine the effects of the COVID-19 epidemic on physical activity and perceived physical fitness in Japanese adults aged 40 to 69 years. Data were collected from an online survey conducted between October 19 and 28, 2020. The analytic sample consisted of 1989 Japanese adults (mean age, 50.1 ± 6.9 years; women, 38.9%) who were aged between 40 and 69 years and completed the online survey. Overall, the PA time per week decreased by 32.4% between October 2019 and April 2020. A decrease in PA time was recorded in October 2020; however, a decline of 15.5% was observed. Compared to individuals who did not perceive a decline in physical fitness, individuals who perceived declining physical fitness during the COVID-19 state of emergency demonstrated a greater decrease in PA time in April 2020 (–50.5%), and this trend continued into October 2020 (–25.0%). These findings may indicate that Japanese adults aged 40 to 69 years who perceived declining physical fitness experienced a greater decrease in physical activity.