Attention mechanism (AM) is a widely used method for improving the performance of convolutional neural networks (CNNs) on computer vision tasks. Despite its pervasiveness, we have a poor understanding of what its effectiveness stems from. It is popularly believed that its effectiveness stems from the visual attention explanation, i.e., attention weights indicate the importance of feature and AM advocates focusing on the important part of an input image rather than ingesting the entire input. However, we find only a weak consistency exists between the attention weights of features and their importance. We verify the feature map multiplication that brings about high-order non-linearity into CNNs is crucial for the effectiveness of AM. Furthermore, we show an essential impact of feature map multiplication on the learned surfaces of CNNs. With the high-order non-linearity, feature map multiplication plays a regularization role on CNNs, which makes the learned curves smoother and more stable in-between real samples (test/training samples in datasets). Thus, compared to vanilla CNNs, CNNs equipped with AM are more robust to noises and yield smaller model sensitivity scores, which is the reason for their better performance.

Processing a large number of proposals usually takes a significant proportion of inference time in two-stage object detection methods. Sparse regions with CNN features (Sparse R-CNN) was proposed using a small number of learnable proposals to replace the proposals derived from anchors. To decrease the missing rate, Sparse R-CNN uses six iterative detection heads to gradually regress the detection boxes to the corresponding objects, which hence increases the inference time. To reduce the number of iterative heads, we propose the iterative Hungarian assigner that encourages Sparse R-CNN to generate multiple proposals for each object at the inference stage. This decreases the missing rate when the number of iterative heads is small. As a result, Sparse R-CNN using the proposed assigner needs fewer iterative heads but gives higher detection accuracy. Also, we observe that the multi-layer outputs of the feature pyramid network contribute little to Sparse R-CNN and propose using a single-layer output neck to replace it. The single-layer output neck further improves the inference speed of Sparse R-CNN without the cost of detection accuracy. Experimental results show that the proposed iterative Hungarian assigner together with the single-layer output neck improves Sparse R-CNN by 2.5 AP50 on the Microsoft common objects in context (MS-COCO) dataset and improves Sparse R-CNN by 3.0 AP50 on the PASCAL visual object classes (VOC) dataset while decreasing 30% floating point operations (FLOPs).

The region proposal network is indispensable to two‐stage object detection methods. It generates a fixed number of proposals that are to be classified and regressed by detection heads to produce detection boxes. However, the fixed number of proposals may be too large when an image contains only a few objects but too small when it contains much more objects. Considering this, the authors explored determining a proper number of proposals according to the number of objects in an image to reduce the computational cost while improving the detection accuracy. Since the number of ground truth objects is unknown at the inference stage, the authors designed a simple but effective module to predict the number of foreground regions, which will be substituted for the number of objects for determining the proposal number. Experimental results of various two‐stage detection methods on different datasets, including MS‐COCO, PASCAL VOC, and CrowdHuman showed that equipping the designed module increased the detection accuracy while decreasing the FLOPs of the detection head. For example, experimental results on the PASCAL VOC dataset showed that applying the designed module to Libra R‐CNN and Grid R‐CNN increased over 1.5 AP ₅₀ while decreasing the FLOPs of detection heads from 28.6 G to nearly 9.0 G.