To enhance the accuracy of product recognition in non-store retail sales and address misidentification and missed detection caused by occlusion in densely placed goods, we propose an improved YOLOv8-based network: Dense-YOLO. We first introduce an enhanced multi-scale feature extraction module (EMFE) in the feature extraction layer and employ a lightweight feature fusion strategy (LFF) in the feature fusion layer to improve the network’s performance. Next, to enhance the performance of dense product recognition, particularly when handling small and multi-scale objects in complex settings, we propose a novel multi-scale dynamic interaction attention mechanism (MDIAM). This mechanism combines dynamic channel weight adjustment and multi-scale spatial convolution to emphasize crucial features, while avoiding overfitting and enhancing model generalization. Finally, a cross-layer feature interaction mechanism is introduced to strengthen the interaction between low- and high-level features, further improving the model’s expressive power. Using the public COCO128 dataset and over 2000 daily smart retail cabinet product images compiled in our laboratory, we created a dataset covering 50 product categories for ablation and comparison experiments. The experimental results indicate that the accuracy under MDIAM is improved by 1.6% compared to other top-performing models. The proposed algorithm achieves an mAP of 94.9%, which is a 1.0% improvement over the original model. The enhanced algorithm not only significantly improves the recognition accuracy of individual commodities but also effectively addresses the issues of misdetection and missed detection when multiple commodities are recognized simultaneously.