Ongoing advancements in machine learning show great promise for automating medical data interpretation, potentially saving valuable time in life-threatening situations. One such area is the analysis of electrocardiograms (ECGs). In this study, we investigate the effectiveness of using a DenseNet121 encoder with three decoder architectures: Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), and a Transformer-based approach. We utilize these models to generate automated ECG reports from the publicly available PTB-XL dataset. Our results show that the DenseNet121 encoder paired with a GRU decoder yields higher performance than previously achieved. It achieves a METEOR (Metric for Evaluation of Translation with Explicit Ordering) score of 72.19%, outperforming the previous best result of 55.53% from a ResNet34-based model that used LSTM and Transformer components. We also discuss several important design choices, such as how to initialize decoders, how to use attention mechanisms, and how to apply data augmentation. These findings offer valuable insights into creating more robust and reliable deep learning tools for ECG interpretation.