Most recent multispectral object detectors employ a two-branch structure to extract features from RGB and thermal images. While the two-branch structure achieves better performance than a single-branch structure, it overlooks inference efficiency. This conflict is increasingly aggressive, as recent works solely pursue higher performance rather than both performance and efficiency. In this paper, we address this issue by improving the performance of efficient single-branch structures. We revisit the reasons causing the performance gap between these structures. For the first time, we reveal the information interference problem in the naive early-fusion strategy adopted by previous single-branch structures. Besides, we find that the domain gap between multispectral images, and weak feature representation of the single-branch structure are also key obstacles for performance. Focusing on these three problems, we propose corresponding solutions, including a novel shape-priority early-fusion strategy, a weakly supervised learning method, and a core knowledge distillation technique. Experiments demonstrate that single-branch networks equipped with these three contributions achieve significant performance enhancements while retaining high efficiency. Our code will be available at https://github.com/XueZ-phd/Efficient-RGB-T-Early-Fusion-Detection .

Object detection is of wide application for its capability in recognizing and locating targets in scenes. Under heavy hazy conditions, however, the detection performance on RGB images will greatly degrade since the image contents are polluted. In this work, we propose an imaging system to acquire three-channel images in the shortwave infrared (SWIR) spectrum to facilitate object detection under hazy conditions. The system captures SWIR images in the form of pseudo color that are most suitable for detecting objects, such as pedestrians and vehicles. Two different types of filters, i.e., liquid crystal tunable filter (LCTF) and optical filters, are employed in our imaging system design. We use the LCTF to acquire narrowband hyperspectral images, which are fed into a band simulation model to generate wideband images for optimal band selection. We present a specific measure called recognition and localization (RL) score to evaluate the detection performance of three-band combinations. Based on the measure, optimal bands are determined using an efficient searching algorithm. Then, we customize three optical filters and install them on a filter wheel, with which we can acquire three-channel images in the SWIR spectrum. The effectiveness of our imaging system is evaluated on a self-collected RGB-SWIR image dataset. Experimental results indicate that, compared with the RGB images after haze removal, the three-channel SWIR images acquired by our imaging system are of higher quality and can achieve better object detection performance under hazy conditions.

Tobacco leaf grading plays a crucial role in ensuring the quality of tobacco production. For a very long period, the grading process is manually determined by experienced experts. In recent years, some methods have been introduced to automate the grading process by utilizing the reflection images of tobacco leaves. However, the high visual similarity among reflection images at different grades renders a single reflection image insufficient for achieving accurate grading. Besides, the tobacco leaves with an identical grade may have inconsistent visual appearances due to their different planting locations. It is known that an expert integrates multimodal information such as visual, tactile, and planting location cues when performing grading. Inspired by this, we propose an end-to-end Cross-modal Enhancement Network, named CMENet, for automatic tobacco leaf grading. In addition to the common reflection image, the network also adopts a transmission image to incorporate the thickness information in manual grading. CMENet comprises a difference-aware fusion module and a meta self-attention module, enabling the extraction of multimodal information from the transmission image and the planting location, respectively. Experimental results demonstrate that our CMENet achieves a high grading accuracy (80.15%) when incorporating multimodal information, surpassing the performance of existing methods that rely solely on reflection images.