Background: Physical inactivity affects 66.2% of Chilean adults. School physical education (PE) is the only guaranteed institutional setting for physical activity (PA) for the entire school population; however, evidence suggests that the affective quality of PE experiences—more than their mere quantity—shapes PA attitudes and levels in adulthood. Objective: To synthesize the scientific evidence on the relationship between school PE memories and PA practice in adults, analyzing affective, motivational, and behavioral mediating mechanisms, with emphasis on the Chilean context. Methods: Systematic review with quantitative synthesis following PRISMA 2020. Searches were conducted in PubMed, Scopus, WoS, SciELO, LILACS, and Google Scholar (2010–2026). Methodological quality was assessed with NOS (observational studies), PEDro (intervention designs), and CASP (qualitative/mixed studies). Effect sizes (Pearson's r) with 95% confidence intervals were calculated using a random-effects model. Results: Of 1,284 identified records, 31 primary-data studies met inclusion criteria (18 cross-sectional retrospective, 9 longitudinal, 4 qualitative/mixed; cumulative N ≈ 42,000 participants). PE enjoyment was associated with more favorable PA attitudes (r = 0.37; 95% CI: 0.31–0.43) and lower sedentarism (r = −0.14; 95% CI: −0.21–−0.07). Social exclusion in PE predicted adult inactivity (r = −0.22; 95% CI: −0.30–−0.14). PA tracking from childhood to adulthood was low but significant (r = 0.14–0.19), mediated by motivational variables. Gender differences were consistent: women reported more negative PE experiences and showed greater adult inactivity. Conclusions: School PE memories are a relevant determinant of adult PA, mediated by enjoyment, perceived competence, and autonomous motivation. In Chile, where specific research on this relationship is scarce, these findings justify the pedagogical redesign of PE toward inclusive and meaningful experiences for all students.