Despite significant advances in deep learning-based image captioning, many state-of-the-art models still struggle to balance visual grounding (i.e., accurate object and scene descriptions) with linguistic coherence (i.e., grammatical fluency and appropriate use of non-visual tokens such as articles and prepositions). To address these limitations, we propose a hybrid image captioning framework that integrates handcrafted and deep visual features. Specifically, we combine local descriptors—Scale-Invariant Feature Transform (SIFT) and Bag of Features (BoF)—with high-level semantic features extracted using ResNet50. This dual representation captures both fine-grained spatial details and contextual semantics. The decoder employs Bahdanau attention refined with an Attention-on-Attention (AoA) mechanism to optimize visual-textual alignment, while GloVe embeddings and a GRU-based sequence model ensure fluent language generation. The proposed system is trained on 200,000 image-caption pairs from the MS COCO train2014 dataset and evaluated on 50,000 held-out MS COCO pairs plus the Flickr8K benchmark. Our model achieves a CIDEr score of 128.3 and a SPICE score of 29.24, reflecting clear improvements over baselines in both semantic precision—particularly for spatial relationships—and grammatical fluency. These results validate that combining classical computer vision techniques with modern attention mechanisms yields more interpretable and linguistically precise captions, addressing key limitations in neural caption generation.

Turning visual evidence into coherent sentences depends on how visual and linguistic representations are combined before decoding. We present two decoder-specific generation strategies, CLS-LSTM and CLS-T5, built on a common CLS-guided Swin–BERT multimodal backbone. Swin-Base extracts a $7\times 7$ grid of 49 spatial feature vectors, each with 1024 dimensions, which are mapped to 512 dimensions through a trainable linear projection initialized from principal components fitted exclusively on the training split. Under a leave-one-out training strategy, one caption serves as the generation target, while the remaining four captions are concatenated and encoded by BERT. The final-layer BERT [CLS] representation is used as the query in multihead cross-attention over the projected visual features, producing a global multimodal conditioning representation. Batch-level modality dropout exposes both decoder configurations to the image-only condition used during validation and testing. CLS-LSTM uses the fused representation for global conditioning, while the previous hidden state of its first LSTM layer queries coverage-aware Bahdanau attention at each decoding step. CLS-T5 projects the fused representation to the visual feature dimension, injects it into the projected visual feature sequence, refines the conditioned sequence using a single Transformer encoder layer, and supplies the resulting visual memory directly to the T5-small decoder while bypassing the original T5 encoder. When auxiliary text is unavailable, CLS-LSTM replaces the BERT [CLS] query with a learned null-text query, whereas CLS-T5 bypasses the auxiliary text fusion branch and conditions the visual features using a learned fusion bias. We evaluate both decoder configurations on the Karpathy splits of Flickr30K and MS COCO 2014 under image-only inference. On Flickr30K, across three random seeds, CLS-LSTM achieves mean BLEU-4 and CIDEr scores of $23.97\pm 1.05$ and $50.92\pm 1.08$, respectively, whereas CLS-T5 achieves $27.00\pm 1.09$ and $64.84\pm 2.86$. On MS COCO 2014 for seed 0, CLS-LSTM obtains BLEU-4 and CIDEr scores of 32.16 and 105.18, while CLS-T5 obtains 35.38 and 117.15. These results show that the shared CLS-guided framework supports both recurrent and Transformer-based caption generation while preserving image-only inference.