The inconsistency between classification and regression is a common problem in the field of object detection. Such inconsistency may lead to undetected objects, false detection, and regression boxes overlapping in the detection results. It has been determined that the inconsistency is mainly caused by feature coupling and the lack of information regarding the interactions between detection heads. In this study, the characteristics of spatial invariance were used, and the ability to fit the data distribution was enhanced by fully connected layers. A feature fusion module (FFM) was proposed in order to enhance the capabilities of the model’s feature extractions. This study also further considered the inconsistency between the loss functions and the proposed regression loss function (RMAE) based on mean absolute error (MAE) for the purpose of improving the location quality. Furthermore, in order to solve the problem of the lack of information regarding the interactions between detection heads, an inconsistency loss function (L_in) was added on the basis of the feature fusion module. Then, to evaluate the effectiveness of the proposed methods, the proposed feature fusion network (FMRNet) was trained based on RetinaNet. The experimental results demonstrated that this study’s proposed methods surpassed the accuracy of some existing detectors when FMRNet was adopted. It was confirmed that the proposed methods had the ability to solve the problems of undetected objects, false detection, and regression boxes overlapping.