The detection of concealed suspicious objects in public places is a critical issue and a popular research topic. Terahertz (THz) imaging technology, as an emerging detection method, can penetrate materials without emitting ionizing radiation, providing a new approach to detecting concealed suspicious objects. This study focuses on the detection of concealed suspicious objects wrapped in different materials such as polyethylene and kraft paper, including items like scissors, pistols, and blades, using THz imaging technology. To address issues such as the lack of texture details in THz images and the contour similarity of different objects, which can lead to missed detections and false alarms, we propose a THz concealed suspicious object detection model based on SMR–YOLO (SPD_Mobile + RFB + YOLO). This model, based on the MobileNext network, introduces the spatial-to-depth convolution (SPD-Conv) module to replace the backbone network, reducing computational and parameter load. The inclusion of the receptive field block (RFB) module, which uses a multi-branch structure of dilated convolutions, enhances the network’s depth features. Using the EIOU loss function to assess the accuracy of predicted box localization further optimizes convergence speed and localization accuracy. Experimental results show that the improved model achieved mAP@0.5 and mAP@0.5:0.95 scores of 98.9% and 89.4%, respectively, representing improvements of 0.2% and 1.8% over the baseline model. Additionally, the detection speed reached 108.7 FPS, an improvement of 23.2 FPS over the baseline model. The model effectively identifies concealed suspicious objects within packages, offering a novel approach for detection in public places.