Background: People with physical disabilities and/or chronic diseases report lower levels of physical activity and well-being than the general population, which potentially is exacerbated through the COVID-19 pandemic. This study explored the international literature on physical activity, sedentary behavior and well-being in adults with physical disabilities and/or chronic diseases during the first wave of the pandemic. Method: In a rapid review, we included studies reporting on physical activity, sedentary behavior and/or well-being in adults with physical disabilities and/or chronic diseases. Four databases (Pubmed, CINAHL, PsycInfo, Embase) were searched for studies published until 30 September 2020. Results: We included twenty-nine studies involving eleven different types of disabilities or health conditions from twenty-one different countries. Twenty-six studies reported on physical activity, of which one reported an increase during the COVID-19 pandemic, four studies reported no difference, and twenty-one studies reported a decrease. Thirteen studies reported a decline in well-being. Only one study measured sedentary behavior, reporting an increase. Conclusion: Despite the variety in methods used, almost all studies reported negative impacts on physical activity and well-being in people with physical disabilities and/or chronic disease during the first wave of the pandemic. These findings highlight the importance of supporting this population, especially in times of crisis.