Articulation modeling, feature extraction, and classification are the important components of pedestrian segmentation. Usually, these components are modeled independently from each other and then combined in a sequential way. However, this approach is prone to poor segmentation if any individual component is weakly designed. To cope with this problem, we proposed a spatio-temporal convolutional neural network named PedNet which exploits temporal information for spatial segmentation. The backbone of the PedNet consists of an encoder–decoder network for downsampling and upsampling the feature maps, respectively. The input to the network is a set of three frames and the output is a binary mask of the segmented regions in the middle frame. Irrespective of classical deep models where the convolution layers are followed by a fully connected layer for classification, PedNet is a Fully Convolutional Network (FCN). It is trained end-to-end and the segmentation is achieved without the need of any pre- or post-processing. The main characteristic of PedNet is its unique design where it performs segmentation on a frame-by-frame basis but it uses the temporal information from the previous and the future frame for segmenting the pedestrian in the current frame. Moreover, to combine the low-level features with the high-level semantic information learned by the deeper layers, we used long-skip connections from the encoder to decoder network and concatenate the output of low-level layers with the higher level layers. This approach helps to get segmentation map with sharp boundaries. To show the potential benefits of temporal information, we also visualized different layers of the network. The visualization showed that the network learned different information from the consecutive frames and then combined the information optimally to segment the middle frame. We evaluated our approach on eight challenging datasets where humans are involved in different activities with severe articulation (football, road crossing, surveillance). The most common CamVid dataset which is used for calculating the performance of the segmentation algorithm is evaluated against seven state-of-the-art methods. The performance is shown on precision/recall, F ${}_1$ , F ${}_2$ , and mIoU. The qualitative and quantitative results show that PedNet achieves promising results against state-of-the-art methods with substantial improvement in terms of all the performance metrics.

Capsule endoscopy, which uses a wireless camera to take images of the digestive tract, is emerging as an alternative to traditional colonoscopy. The diagnostic values of these images depend on the quality of revealed underlying tissue surfaces. In this paper, we consider the problem of enhancing the visibility of detail and shadowed tissue surfaces for capsule endoscopy images. Using concentric circles at each pixel for random walks combined with stochastic sampling, the proposed method enhances the details of vessel and tissue surfaces. The framework decomposes the image into two detailed layers that contain shadowed tissue surfaces and detail features. The target pixel value is recalculated for the smooth layer using similarity of the target pixel to neighboring pixels by weighting against the total gradient variation and intensity differences. In order to evaluate the diagnostic image quality of the proposed method, we used clinical subjective evaluation with a rank order on selected KID image database and compared it to state-of-the-art enhancement methods. The result showed that the proposed method provides a better result in terms of diagnostic image quality and objective quality contrast metrics and structural similarity index.

For automating deburring of cast parts, this paper proposes a general method for estimating burr height using 3D vision sensor that is robust to missing data in the scans and sensor noise. Specifically, we present a novel data-driven method that learns features that can be used to align clean CAD models from a workpiece database to the noisy and incomplete geometry of a RGBD scan. Using the learned features with Random sample consensus (RANSAC) for CAD to scan registration, learned features improve registration result as compared to traditional approaches by (translation error ( $$\Delta $$Δ 18.47 mm) and rotation error( $$\Delta 43 ^\circ $$Δ 43 ∘ )) and accuracy(35%) respectively. Furthermore, a 3D-vision based automatic burr detection and height estimation technique is presented. The estimated burr heights were verified and compared with measurements from a high resolution industrial CT scanning machine. Together with registration, our burr height estimation approach is able to estimate burr height similar to high resolution CT scans with Z-statistic value ( $$z=0.279$$z = 0.279 ).

Individual fish identification and recognition is an important step in the conservation and management of fisheries. One of most frequently used methods involves capturing and tagging fish. However, these processes have been reported to cause tissue damage, premature tag loss, and decreased swimming capacity. More recently, marine video recordings have been extensively used for monitoring fish populations. However, these require visual inspection to identify individual fish. In this work, we proposed an automatic method for the identification of individual brown trouts, Salmo trutta. We developed a deep convolutional architecture for this purpose. Specifically, given two fish images, multi-scale convolutional features were extracted to capture low-level features and high-level semantic components for embedding space representation. The extracted features were compared at each scale for capturing representation for individual fish identification. The method was evaluated on a dataset called NINA204 based on 204 videos of brown trout and on a dataset TROUT39 containing 39 brown trouts in 288 frames. The identification method distinguished individual fish with 94.6% precision and 74.3% recall on a NINA204 video sequence with significant appearance and shape variation. The identification method takes individual fish and is able to distinguish them with precision and recall percentages of 94.6% and 74.3% on NINA204 for a video sequence with significant appearance and shape variation.

Manipulation tasks on subsea instalments require extremely precise detection and localization of objects of interest. This problem is referred to as “pose estimation”. In this work, we present a framework for detecting and predicting 6DoF pose for relevant objects (fish-tail, gauges, and valves) on a subsea panel under varying water turbidity. A deep learning model that takes 3D vision data as an input is developed, providing a more robust 6D pose estimate. Compared to the 2D vision deep learning model, the proposed method reduces rotation and translation prediction error by ($-\Delta {0.39}^{\circ }$) and translation ($-\Delta 6.5$ mm), respectively, in high turbid waters. The proposed approach is able to provide object detection as well as 6D pose estimation with an average precision of 91%. The 6D pose estimation results show ${2.59}^{\circ }$ and $6.49$ cm total average deviation in rotation and translation as compared to the ground truth data on varying unseen turbidity levels. Furthermore, our approach runs at over 16 frames per second and does not require pose refinement steps. Finally, to facilitate the training of such model we also collected and automatically annotated a new underwater 6D pose estimation dataset spanning seven levels of turbidity.

Capsule video endoscopy, which uses a wireless camera to visualize the digestive tract, is emerging as an alternative to traditional colonoscopy. Colonoscopy is considered as the gold standard for visualizing the colon and takes 30 frames per second. Capsule images, on the other hand, are taken with low frame rate (average five frames per second), which makes it difficult to find pathology and results in eye fatigue for viewing. In this paper, we propose a variational algorithm to smooth the video temporally and create a visually pleasant video. The main objective of the paper is to increase the frame rate to be closer to that of the colonoscopy. We propose variational energy that takes into consideration both motion estimation and intermediate frame intensity interpolation using the surrounding frames. The proposed formulation incorporates both pixel intensity and texture feature in the optical flow objective function such that the interpolation at the intermediate frame is directly modeled. The main feature of this formulation is that error in motion estimation is incorporated in our model, so that only robust motion estimation are used in estimating the intensity of the intermediate frame. We derived Euler-Lagrange equations and showed an efficient numerical scheme that can be implemented on graphics hardware. Finally, a motion compensated frame rate doubling version of our method is implemented. We evaluate the quality of both 90 and 100% of the frames for medical diagnosis domain through objective image quality metrics. Our method improves state-of-the-art result for 90% frames while performing equivalent for the remaining cases with other existing methods. In the last section, we show application of frame interpolation to informative frame segment visualization and to reduce the power consumption.