Background: This 12-year study aimed to compare the longitudinal change in left ventricular diastolic dysfunction (LVDD) between healthy elderly, coronary artery disease (CAD), and hypertension (HTN) patients. Methods: From 2008 to 2020, 1476 patients were included, and 3181 echocardiography examinations were conducted. Finally, 130 participants (36 healthy elderly (79.39 ± 9.51 years old), 51 with CAD (68.31 ± 12.09 years old), and 43 with HTN (68.31 ± 12.09 years old)) with more than a 10-year follow-up period were included in the final analysis. Results: The change in diastolic function was different among these subjects according to the integrated score index (elderly vs. HTN, p = 0.01; CAD vs. HTN, p = 0.01), septal E/e′ ratio (elderly vs. HTN, p < 0.001; CAD vs. HTN, p = 0.01), lateral E/e′ ratio (elderly vs. HTN, p < 0.001; CAD vs. HTN, p < 0.001), and NYHA functional class (elderly vs. HTN, p = 0.03; CAD vs. HTN, p < 0.001). Additionally, per one-year increase in age, the integrated score index increased 0.2 in the healthy elderly, 0.15 in the CAD, and 0.06 in the HTN patients (all p < 0.05). Conclusion: Under aggressive treatment, diastolic function was relatively preserved in HTN subjects with aging in comparison with elderly and CAD subjects.