Anthracycline-based regimens with or without anti-human epidermal growth factor receptor (HER) 2 agents such as trastuzumab are effective in breast cancer treatment. Nevertheless, heart failure (HF) has become a significant side effect of these regimens. This study aimed to investigate the incidence and factors associated with HF in breast cancer patients treated with anthracyclines with or without trastuzumab. A retrospective cohort study was performed in patients with breast cancer who were treated with anthracyclines with or without trastuzumab between 1 January 2014 and 31 December 2018. The primary outcome was the incidence of HF. The secondary outcome was the risk factors associated with HF by using the univariable and multivariable cox-proportional hazard model. A total of 475 breast cancer patients were enrolled with a median follow-up time of 2.88 years (interquartile range (IQR), 1.59–3.93). The incidence of HF was 3.2%, corresponding to an incidence rate of 11.1 per 1000 person-years. The increased risk of HF was seen in patients receiving a combination of anthracycline and trastuzumab therapy, patients treated with radiotherapy or palliative-intent chemotherapy, and baseline left ventricular ejection fraction <65%, respectively. There were no statistically significant differences in other risk factors for HF, such as age, cardiovascular comorbidities, and cumulative doxorubicin dose. In conclusion, the incidence of HF was consistently high in patients receiving combination anthracyclines trastuzumab regimens. A reduced baseline left ventricular ejection fraction, radiotherapy, and palliative-intent chemotherapy were associated with an increased risk of HF. Intensive cardiac monitoring in breast cancer patients with an increased risk of HF should be advised to prevent undesired cardiac outcomes.