Historical records of growth rates of the key Caribbean reef framework-building coral Orbicella faveolata can be fundamental not only to understand how these organisms respond to environmental changes but also to infer future responses of reef ecosystems in a changing world. While coral growth rates have been widely documented throughout the Caribbean, the drivers of coral growth variability remain poorly understood. Here, we provide a record spanning 53 years (1963–2015) of the coral growth parameters for five O. faveolata core samples collected at Serrana Atoll, inside the Seaflower Biosphere Reserve, Colombian Caribbean. Coral cores were extracted from reefs isolated from direct anthropogenic impacts, and growth estimations (skeletal density, linear extension, and calcification rates) were derived using computerized tomography. Master records of coral growth parameters were evaluated to identify long-term trends and to relate growth responses with sea surface temperature (SST), the Atlantic Multi-decadal Oscillation (AMO), North Atlantic Oscillation (NAO) and Southern Oscillation indexes, aragonite saturation state (Ω_arag), and degree heating months (DHM). We found significant negative relationships between density and mean SST, maximum SST, AMO, and DHM. Moreover, density showed significant positive correlations with NAO and Ω_arag. Extension rate did not show significant correlations with any environmental variable. However, there were significant negative correlations between calcification and maximum SST, AMO, and DHM. Trends of coral growth indicated a significant reduction in density and calcification over time, which were best explained by changes in Ω_arag. Inter-annual declines in calcification and density up to 25% (relative to historical mean) were associated to the impacts of previously recorded mass bleaching events (1998, 2005, and 2010). Our study provides further evidence that AMO and Ω_arag are important drivers affecting coral growth rates in the Southwestern Caribbean. Therefore, we suggest upcoming variations of AMO and future trajectories of Ω_arag in the Anthropocene could have a substantial influence on future disturbances, ecological process and responses of the Caribbean reefs.