Weeds are among the major factors that could harm crop yield. Site-specific weed management has become an effective tool to control weed and machine vision combined with image processing is an effective approach for weed detection. In this work, an encoder-decoder deep learning network was investigated for pixel-wise semantic segmentation of crop and weed. Different input representations including different color space transformations and color indices were compared to optimize the input of the network. Three image enhancement methods were investigated to improve model robustness against different lighting conditions. The results show that for images without enhancement, color space transformation and vegetation indices without NIR (Near Infrared) information did not improve the segmentation results, while inclusion of NIR information significantly improved the segmentation accuracy, indicating the effectiveness of NIR information for precise segmentation under weak lighting condition. Image enhancement improved the image quality and consequently the robustness of segmentation models against different lighting conditions. The best MIoU value for pixel-wise segmentation was 88.91% and the best mean accuracy of object-wise segmentation was 96.12%. The deep network and image enhancement methods applied in this work provided promising segmentation results for weed detection and did not need large amount of data for model training, which is suitable for site-specific weed management.

Diabetic retinopathy is one of the main causes of blindness in human eyes, and lesion segmentation is an important basic work for the diagnosis of diabetic retinopathy. Due to the small lesion areas scattered in fundus images, it is laborious to segment the lesion of diabetic retinopathy effectively with the existing U‐Net model. In this paper, we proposed a new lesion segmentation model named FFU‐Net (Feature Fusion U‐Net) that enhances U‐Net from the following points. Firstly, the pooling layer in the network is replaced with a convolutional layer to reduce spatial loss of the fundus image. Then, we integrate multiscale feature fusion (MSFF) block into the encoders which helps the network to learn multiscale features efficiently and enrich the information carried with skip connection and lower‐resolution decoder by fusing contextual channel attention (CCA) models. Finally, in order to solve the problems of data imbalance and misclassification, we present a Balanced Focal Loss function. In the experiments on benchmark dataset IDRID, we make an ablation study to verify the effectiveness of each component and compare FFU‐Net against several state‐of‐the‐art models. In comparison with baseline U‐Net, FFU‐Net improves the segmentation performance by 11.97%, 10.68%, and 5.79% on metrics SEN, IOU, and DICE, respectively. The quantitative and qualitative results demonstrate the superiority of our FFU‐Net in the task of lesion segmentation of diabetic retinopathy.

The Hyperspectral Image (HSI) classification aims to assign each pixel to a land cover category. It is receiving increasing attention from both industry and academia. The main challenge lies in capturing reliable and informative spatial and spectral dependencies concealed in the HSI for each class. To address the challenge, we propose a Spatial-Spectral 1DSwin Transformer with Group-wise Feature Tokenization (SS1DSwin) for HSI classification. Specifically, we reveal local and hierarchical spatial-spectral relationships from two different perspectives. It mainly consists of a Group-wise Feature Tokenization Module (GFTM) and a 1DSwin Transformer with Cross-block Normalized Connection Module (TCNCM). For GFTM, we reorganize an image patch into overlapping cubes, and further generate group-wise token embeddings with Multi-head Self-Attention (MSA) to learn the local spatial-spectral relationship along the spatial dimension. For TCNCM, we adopt the shifted windowing strategy when acquiring the hierarchical spatial-spectral relationship along the spectral dimension with 1D Window based Multi-head Self-Attention (1DW-MSA) and 1D Shifted Window based Multi-head Self-Attention (1DSW-MSA), and leverage Cross-block Normalized Connection (CNC) to adaptively fuse the feature maps from different blocks. In SS1DSwin, we apply these two modules in order and predict the class label for each pixel. To test the effectiveness of the proposed method, extensive experiments are conducted on four HSI datasets, and the results indicate that SS1DSwin outperforms several current state-of-the-art methods. The source code of the proposed method is available at https://github.com/Minato252/SS1DSwin.

Video summarization attempts to create concise and complete synopsis of a video through identifying the most informative and explanatory parts while removing redundant video frames, which facilitates retrieving, managing and browsing video efficiently. Most existing video summarization approaches heavily rely on enormous high-quality human-annotated labels or fail to produce semantically meaningful video summaries with the guidance of prior information. Without any supervised labels, we propose Self-supervised Adversarial Video Summarizer (2SAVS) that exploits context latent sequence learning to generate satisfying video summary. To implement it, our model elaborates a novel pretext task of identifying latent sequences and normal frames by training self-supervised generative adversarial network (GAN) with several well-designed losses. As the core components of 2SAVS, Clip Consistency Representation (CCR) and Hybrid Feature Refinement (HFR) are developed to ensure semantic consistency and continuity of clips. Furthermore, a novel separation loss is designed to explicitly enlarge the distance between prediction frame scores to effectively enhance the model’s discriminative ability. Differently, latent sequences, additional finetune operations and generators are not required when inferring video summary. Experiments on two challenging and diverse datasets demonstrate that our approach outperforms other state-of-the-art unsupervised and weakly-supervised methods, and even produces comparable results with several excellent supervised methods.

As a detection tool to identify metal or alloy, metallographic quantitative analysis has received increasing attention for its ability to evaluate quality control and reveal mechanical properties. The detection procedure is mainly operated manually to locate and characterize the constitution in metallographic images. The automatic detection is still a challenge even with the emergence of several excellent models. Benefiting from the development of deep learning, with regard to two different metallurgical structural steel image datasets, we propose two attention-aware deep neural networks, Modified Attention U-Net (MAUNet) and Self-adaptive Attention-aware Soft Anchor-Point Detector (SASAPD), to identify structures and evaluate their performance. Specifically, in the case of analyzing single-phase metallographic image, MAUNet investigates the difference between low-frequency and high-frequency and prevents duplication of low-resolution information in skip connection used in an U-Net like structure, and incorporates spatial-channel attention module with the decoder to enhance interpretability of features. In the case of analyzing multi-phase metallographic image, SASAPD explores and ranks the importance of anchor points, forming soft-weighted samples in subsequent loss design, and self-adaptively evaluates the contributions of attention-aware pyramid features to assist in detecting elements in different sizes. Extensive experiments on the above two datasets demonstrate the superiority and effectiveness of our two deep neural networks compared to state-of-the-art models on different metrics.

With the advancement of remote sensing technology, more and more modalities are becoming available for land cover classification tasks, helping to address the issue of insufficiency and incompleteness caused by modeling on single-source remote sensing images. However, most existing remote sensing data classifiers are struggling to capture reliable and informative spatial and spectral dependencies and neglect the correlations and complementarity between different modalities. To address the challenges, we propose the SpatialSpectral Residual Cross-Attention Transformer (S2RCFormer) for Multimodal Remote Sensing Data Classification (MRSDC). It mainly consists of Patch-wise Convolutional Module (PTConv), Pixel-wise Convolutional Module (PXConv), Residual Cross-Attention Tokenization Module (RCTM) and Transformer Feature Fusion Module (TFFM). To make full use of multimodal cues, PTConv extracts patch-wise spatial-spectral and spatial feature from HSI and other modality, respectively, while PXConv attempts to exploit detailed spectral feature from HSI and unique pixel-wise feature from other modality. Afterwards, RCTM is employed to leverage one-dimensional Residual Cross-Attention for adaptively fusing diverse features across different modalities in a tokenization fashion. As for TFFM, it establishes long-range dependencies on the set of heterogeneous tokens followed by a Multiple Layer Perceptron (MLP) for predicting the output category. To verify the effectiveness of the proposed method, extensive experiments are conducted on three benchmark datasets (Trento, MUUFL, Augsburg) using four different modality combinations. The results indicate that the proposed approach shows comparable results to other state-of-the-art methods over different metrics. The source code of S2RCFormer is available at https://github. com/terayco/S2RCFormer.