Background: The psychological impact of COVID-19 is multifaceted, both acute and chronic, and has not affected everyone equally. Method: This longitudinal study compared those with and without Adverse Childhood Experiences (ACEs) on measures of psychological distress and wellbeing over time. Results: All groups (No ACE, Low ACE, and High ACE) had similar levels of distress at Time 1, with significant increases in psychological distress for those with ACEs over time, but not for those without. Psychological Flexibility was strongly and significantly associated with decreases in psychological distress and improved wellbeing. It significantly mediated the relationship between ACE and wellbeing. Conclusions: Those with ACEs report significantly increased psychological distress over time, compared to those without ACE during the COVID-19 pandemic. Evidence-based interventions using Psychological Flexibility may improve mental health and wellbeing to help further mediate its effects.