25 July 2024 in IEEE Transactions on Circuits and Systems for Video Technology
An automatic vision-based sewer inspection plays a vital role of sewage system in a modern city. Recent advances focus on modeling a deep learning-based method to realize the sewer inspection system, benefiting from the capability of data-driven feature extraction. Although the acceptable performances of sewer defect classification are achieved, there is still a gap between the emerged methods and actual application scenarios. The first issue is that the multi-focus complementarity is ignored to represent the sewer defect, resulting in capturing the multi-scale information of sewer defect inefficiently. Second, the inherent uncertainty of sewer defect is not considered, while the serious unknown sewer defect categories would be missed, resulting in the untrustworthy sewer inspection. In this paper, we focus on quick-view (QV)-based sewer inspection, while a trustworthy multi-focus fusion framework (TMFF) is proposed, jointly combining multi-label classification and uncertainty estimation. Specifically, focal segment module (FSM) is designed based on optical flow to split the QV sewer video into long-focus and short-focus segments, where the multi-focus segments can be modeled to represent the multi-scale information of sewer defect. Then, evidential deep learning (EDL) is introduced to quantify the uncertainty, while joint expert scheme (JES) is designed to aggregate the expert opinions of multi-focus segments. Moreover, evidential disambiguating strategy (EDS) is proposed to alleviate the ambiguity of uncertainty estimation. Extensive experiments are conducted on VideoPipe, in which the superiority of TMFF is demonstrated compared with the state-of-the-art methods. Furthermore, we validate the potential capability of TMFF against the unknown cases of sewer defects.
28 February 2023 in IEEE Transactions on Instrumentation and Measurement
An automatic vision-based sewer inspection plays a key role of sewage system in a modern city. Recent advances focus on utilizing a deep learning model to realize the sewer inspection system, benefiting from the capability of data-driven feature extraction. However, the ambiguity of sewer defects in the feature space is ignored, deteriorating the performance of sewer inspection. There are two reasons for such ambiguity. First, the defect-irrelevant region interferes the feature extraction of the model. Second, the setting of multilabel is an inherent challenge of extracting discriminative feature for different defect classes from defect-relevant region. In this article, we propose a multilabel sewer defect classification method, which can purify the sewer defect components in latent spaces, thereby mitigating the ambiguity of sewer defects. Specifically, a novel self-purification module (SPM) is modeled to disentangle the ambiguity of sewer defect feature, which consists of intraclass purification (ICP) and interclass decorrelation (ICD). ICP utilizes the task-aware information to purify the sewer features, and ICD aims to eliminate the cross correlation and defect-irrelevant components simultaneously. Moreover, to ensure the reliability of SPM, center global alignment (CGA) is introduced to avoid the trivial solution. Experimental results demonstrate the superiority of the proposed method compared with seven state-of-the-art methods on the latest benchmark Sewer-ML. The proposed method outperforms the others approximately 8% $\mathrm {F2_{\text {CIW}}}$ with a tolerant inference speed. Finally, the robustness of the proposed method is verified in two perturbed scenarios, where the reliable performance can be guaranteed against the limited perturbations.
23 August 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Subtle variations are invisible to the naked eyes in human physiological signals can reflect important biological and health indicators. Although numerous computer vision methods have been proposed to recover and magnify these changes, most of them either only focus on identifying and recognizing explicit features such as shapes and textures, or are weak in long-term temporal modeling and spatiotemporal interactive perception of implicit biometrics. Therefore, it is difficult for them to robustly overcome various disturbances that affect detection performance. To address these issues, this paper presents TranPhys, a novel remote photoplethysmography (rPPG) network for facial video-based heart rate estimation. Specifically, first, we argue that facial subregions vary over time due to their biological personalities. So we split the input face video into multiple spatiotemporal tubes, build the 3D vision transformer with encoders and decoders to adequately model the high-dimensional representations of the respective regulars in each subregion, and globally coordinate their feedback on the cardiac pulsing waveform. Second, we design the temporal pooling attention to more finely mine the subtle changes hidden in the skin color over time and their long-term contextual rhythm cues. Third, we leverage the self-supervised masked autoencoding paradigm to overcome redundancy to enhance the robustness of our model, and construct the targeted spatiotemporal sampling maps instead of raw input sequences as the pretrained constraint labels to fully inspire self-supervision. We train, validate, and practice our TranPhys on multiple public datasets to demonstrate that our method achieves the competitive performance in remote heart rate estimation.
1 September 2020 in Signal Processing: Image Communication
15 October 2024 in IEEE Signal Processing Letters
An automatic vision-based sewer inspection plays a vital role of sewage system in a modern city. Existing methods have utilized evidential deep learning to construct trusted models. Although the acceptable performance has been achieved in sewer defect classification, the fine-grained information of sewer defects in videos is ignored. Meanwhile, the trade-off between multi-label classification and uncertainty estimation remains challenging. In this paper, support clip-based pareto-optimal evidential network (POEN) is proposed for trusted video-based sewer inspection. Specifically, support clip module (SCM) is designed to capture the fine-grained visual representation of defects from local scale segments. Then, evidential deep learning is introduced to quantify the uncertainty for out-of-distribution detection. Furthermore, Pareto-optimal weighting scheme (PWS) is designed to solve the common trade-off dilemma in multi-task learning. Extensive experiments are conducted on VideoPipe, in which the superiority of POEN is demonstrated compared with the state-of-the-art methods.
11 August 2025 in IEEE Transactions on Artificial Intelligence
In the community of sewer pipe defect analysis, CNN-based methods have suffered from modeling the long-range dependencies in the long-sequence videos, such as closed-circuit television (CCTV-) and quick-view (QV)-based videos, resulting in the suboptimal performance. Besides, Transformer-based methods are limited in the real-time applications due to their high computational complexity during the inference. Recently, Mamba-based methods have performed the significant capability with a trade-off between accuracy and efficiency, which present a promising direction for modeling an efficient defect inspection method. Nevertheless, directly adapting Mamba to sewer pipe defect inspection achieves a poor performance, since the multi-focus complementarity of representing the sewer defects is ignored, and the bidirectional scan method disrupts the spatial and semantic continuity of the sewer pipes. To address these challenges, we propose a method for the video-based sewer pipe defect classification using a state space model, namely PipeMamba. Specifically, a flow-based selection module (FSM) is designed to split the sewer video into the long-focus and shortfocus segments, where the multi-focus segments can be modeled to represent the multi-scale information of sewer defects. Then, a novel circle scan strategy (CSS) is proposed to specify the appearances of the sewer pipes, which aims to capture the sewer defects efficiently without disrupting the semantic consistency and spatial continuity of the sewer pipes. Extensive experiments conducted on the public benchmark demonstrate the superiority of PipeMamba, which achieves the inference speed nearly 3 to 7 times faster than the Transformer-based methods, while maintaining high classification performance. Code is available at: https://anonymous.4open.science/r/PipeMamba-3637.