Plankton, including phytoplankton and zooplankton, are the main source of food for organisms in the ocean and form the base of marine food chain. As the fundamental components of marine ecosystems, plankton is very sensitive to environment changes, and the study of plankton abundance and distribution is crucial, in order to understand environment changes and protect marine ecosystems. This study was carried out to develop an extensive applicable plankton classification system with high accuracy for the increasing number of various imaging devices. Literature shows that most plankton image classification systems were limited to only one specific imaging device and a relatively narrow taxonomic scope. The real practical system for automatic plankton classification is even non-existent and this study is partly to fill this gap. Inspired by the analysis of literature and development of technology, we focused on the requirements of practical application and proposed an automatic system for plankton image classification combining multiple view features via multiple kernel learning (MKL). For one thing, in order to describe the biomorphic characteristics of plankton more completely and comprehensively, we combined general features with robust features, especially by adding features like Inner-Distance Shape Context for morphological representation. For another, we divided all the features into different types from multiple views and feed them to multiple classifiers instead of only one by combining different kernel matrices computed from different types of features optimally via multiple kernel learning. Moreover, we also applied feature selection method to choose the optimal feature subsets from redundant features for satisfying different datasets from different imaging devices. We implemented our proposed classification system on three different datasets across more than 20 categories from phytoplankton to zooplankton. The experimental results validated that our system outperforms state-of-the-art plankton image classification systems in terms of accuracy and robustness. This study demonstrated automatic plankton image classification system combining multiple view features using multiple kernel learning. The results indicated that multiple view features combined by NLMKL using three kernel functions (linear, polynomial and Gaussian kernel functions) can describe and use information of features better so that achieve a higher classification accuracy.

The booming self-driving cars need to understand the behaviors of other road users for better performance. Recognizing pedestrians’ crossing intentions is one of the most critical capabilities of self-driving cars to guarantee safe operations in the urban environment. Some researchers try to predict the future trajectories of pedestrians to avoid a potential collision. Others extract skeleton features from pedestrians to detect specific actions related to the crossing intention. All these methods either neglect the abundant appearance information of pedestrians, or work poorly under severe conditions, e.g., pedestrians standing far away, dim light, and occlusions. In this work, a pedestrian crossing intention recognition (PCIR) framework is proposed to recognize pedestrians’ crossing intention. A module for searching targets of interest is introduced to find the pedestrians who are likely to cross the streets, and perform scene perception simultaneously. An action recognition module utilizes a 3D convolutional neural network (CNN) to automatically extract spatiotemporal features that imply early actions before crossing, like limb movements. A distance encoding module makes full use of the contextual cues, e.g., distances between pedestrians and the oncoming vehicle, local traffic scenes around pedestrians, and vehicle speeds, to improve the recognition accuracy obtained from the action recognition module only. Finally, the PCIR framework fuses these factors to predict pedestrians’ crossing intentions by minimizing a focal loss. Experimental evaluations verify the effectiveness of the proposed method in recognizing pedestrians’ crossing intentions. Comparisons with the skeleton-based methods reveal the robustness of the PCIR framework in the urban environment.

We propose an anomaly detection approach by learning a generative model of moving pedestrians to guarantee public safety. To resolve the existing challenges of anomaly detection in complicated definitions, complex backgrounds, and local occurrence, a weighted convolutional autoencoder-long short-term memory network is proposed to reconstruct raw data and their corresponding optical flow and then perform anomaly detection based on reconstruction errors. Unlike equally treating raw data and optical flow, a novel two-stream framework is proposed to take the reconstructed optical flow as supplementary cues that encode pedestrian motions. A weighted Euclidean loss is proposed to enable the network to concentrate on moving foregrounds, thus restraining background influence. Global-local analysis is used to jointly detect and localize local anomaly in reconstructed raw data. Final detection of anomalous events is achieved by jointly considering the results of the global-local analysis and reconstructed optical flow. Qualitative evaluations verify the effectiveness of our two-stream framework, the weighted Euclidean loss, and the global-local analysis. Moreover, comparisons with state-of-the-art approaches indicate the superiority of our approach in anomaly detection.

Logical stochastic resonance (LSR), the phenomenon in which the interplay of noise and nonlinearity can raise the accurate probability of response to feeble input signals, is studied in this Lettter to extract objects from highly degraded underwater images. Images captured under water, especially in the turbid areas, always suffer from interference through heavy noise caused by the suspended particles. Inherent noise and nonlinearity cause difficulty in processing these images through conventional image processing methods. However, LSR can optimally address such issues. A heavily degraded image is first extended to a 1D form in the direction determined by the illumination condition, and then normalized to be placed in the LSR system as an input signal. Additional Gaussian noise is added to the system as the auxiliary power to help separate the object and the background. Results in the natural offshore area demonstrate the effect of LSR on image processing, and the proposed method creates an interesting direction in the processing of heavily degraded images.

Plankton size spectra monitoring is crucial for managing and conserving aquatic ecosystems. Thus, we develop an in situ size spectra monitoring system to obtain the size spectra of plankton and the information of their living status underwater. The system consists of an imaging unit and an information processing unit. The imaging part applies a darkfield illumination to enhance the image contrast. Three lenses with different magnifications are alternated by a motor automatically to capture sizes of plankton from $3\,\mu$ m to $3\,\text{mm}$ . Moreover, the system can analyze the captured images in real time using the proposed multitask size spectra convolutional neural network, obtaining size spectra and density distribution of plankton simultaneously. Field test confirms that our system performs well both in imaging and information processing. Furthermore, the system can provide the living behavior of plankton, thereby helping biologists to study the aquatic ecosystem effectively and precisely.