The ability of snapshot compressive imaging (SCI) systems to efficiently capture high-dimensional (HD) data has led to an inverse problem, which consists of recovering the HD signal from the compressed and noisy measurement. While reconstruction algorithms grow fast to solve it with the recent advances of deep learning, the fundamental issue of accurate and stable recovery remains. To this end, we propose deep equilibrium models (DEQ) for video SCI, fusing data-driven regularization and stable convergence in a theoretically sound manner. Each equilibrium model implicitly learns a nonexpansive operator and analytically computes the fixed point, thus enabling unlimited iterative steps and infinite network depth with only a constant memory requirement in training and testing. Specifically, we demonstrate how DEQ can be applied to two existing models for video SCI reconstruction: recurrent neural networks (RNN) and Plug-and-Play (PnP) algorithms. On a variety of datasets and real data, both quantitative and qualitative evaluations of our results demonstrate the effectiveness and stability of our proposed method. The code and models are available at: https://github.com/IndigoPurple/DEQSCI.
Traffic prediction, as a core task to alleviate urban congestion and optimize the transport system, has limitations in the integration of multimodal data, making it difficult to comprehensively capture the complex spatio-temporal characteristics of the transport system. Although some studies have attempted to introduce multimodal data, they mostly rely on resource-intensive deep neural network architectures, which have difficultly meeting the demands of practical applications. To this end, we propose a traffic prediction framework based on simple machine learning techniques that effectively integrates property features, amenity features, and emotion features (PAE features). Validated with large-scale real datasets, the method demonstrates excellent prediction performance while significantly reducing computational complexity and deployment costs. This study demonstrates the great potential of simple machine learning techniques in multimodal data fusion, provides an efficient and practical solution for traffic prediction, and offers an effective alternative to resource-intensive deep learning methods, opening up new paths for building scalable traffic prediction systems.
17 October 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Neuromorphic imaging is an emerging technique that imitates the human retina to sense variations in dynamic scenes. It responds to pixel-level brightness changes by asynchronous streaming events and boasts microsecond temporal precision over a high dynamic range, yielding blur-free recordings under extreme illumination. Nevertheless, this modality falls short in spatial resolution and leads to a low level of visual richness and clarity. Pursuing hardware upgrades is expensive and might cause compromised performance due to more burdens on computational requirements. Another option is to harness offline, plug-in-play super-resolution solutions. However, existing ones, which demand substantial sample volumes for lengthy training on massive computing resources, are largely restricted by real data availability owing to the current imperfect high-resolution devices, as well as the randomness and variability of motion. To tackle these challenges, we introduce the first self-supervised neuromorphic super-resolution prototype. It can be self-adaptive to per input source from any low-resolution camera to estimate an optimal, high-resolution counterpart of any scale, without the need of side knowledge and prior training. Evaluated on downstream tasks, such a simple yet effective method can obtain competitive results against the state-of-the-arts, significantly promoting flexibility but not sacrificing accuracy. It also delivers enhancements for inferior natural images and optical micrographs acquired under non-ideal imaging conditions, breaking through the limitations that are challenging to overcome with frame-based techniques. In the current landscape where the use of high-resolution cameras for event-based sensing remains an open debate, our solution is a cost-efficient and practical alternative, paving the way for more intelligent imaging systems.
16 August 2024 in IEEE Transactions on Image Processing
Light field (LF) images enable numerous applications due to their ability to capture information for multiple views. Semantic segmentation is an essential task for LF scene understanding. However, existing supervised methods heavily rely on a large number of pixel-wise annotations. To relieve this problem, we propose a semi-supervised LF semantic segmentation method that requires only a small subset of labeled data and harnesses the LF disparity information. First, we design an unsupervised disparity estimation network, which can determine the disparity map for every view. With the estimated disparity maps, we generate pseudo-labels along with their weight maps for the peripheral views when only the labels of central views are available. We then merge the predictions from multiple views to obtain more reliable pseudo-labels for unlabeled data, and introduce a disparity-semantics consistency loss to enforce structure similarity. Moreover, we develop a comprehensive contrastive learning scheme that includes a pixel-level strategy to enhance feature representations and an object-level strategy to improve segmentation for individual objects. Our method demonstrates state-of-the-art performance on the benchmark LF semantic segmentation dataset under a variety of training settings and achieves comparable performance to supervised methods when trained under 1/2 protocol.
18 June 2019 in Experimental and Therapeutic Medicine
Fetal hydronephrosis (HY) is a frequent congenital condition, which may be detected by prenatal ultrasound. Society for Fetal Urology (SFU) and anterior‑posterior diameter (APD) grading are two major grading systems based on ultrasonography. The present study aimed to assess the predictive value of the SFU and APD grades in patients with fetal HY. A total of 162 patients with 234 kidneys affected by HY were included in the present study. The SFU and APD grades were determined from the ultrasound images at 38 gestational weeks, and a 12‑month follow‑up was performed after birth. The associations of the SFU and APD grades with the outcome of fetal HY, including HY regression, and post‑partum surgery were examined. In the present study, 16 patients with 17 kidneys were diagnosed with pathological HY, and stenosis at the ureteropelvic junction was demonstrated to be a leading cause of pathological HY. Among the 234 kidneys affected by HY, 161 kidneys were scored as SFU grade I, 57 as SFU grade II, 7 as SFU grade III and 9 kidneys as SFU grade IV. According to the APD grading system, 112 kidneys were determined as having low, 104 as having moderate and 18 as having severe HY. The SFU and APD grades were demonstrated to be independently associated with the occurrence of pathological HY by logistic regression analysis with a high diagnostic accuracy to distinguish pathological and physiological HY cases as evidenced by the results of ROC analysis. In addition, univariate and multivariate logistic regression analysis indicated that patients with spontaneous HY regression usually had low SFU and APD grades. Furthermore, the rate of surgery was increased in the group of patients with high SFU or APD grades, and these two systems were identified as independent predictors for the requirement of surgery by Kaplan‑Meier analysis. Patients with pathological HY had high SFU and APD grades, and these two grading systems may be used as reliable predictors for the outcome of fetal HY, including HY regression, and post‑partum surgery.
We present a compact, cost-effective, and highly accurate dispersion-based four-dimensional (4D) feature learning (3D spatial and 1D spectral) method for real-time monitoring of water-suspended microparticles, requiring neither sample preparation nor explicit reconstruction. By jointly leveraging digital holography and spectral imaging, the system uses a single prism to compress holograms with transmission spectral information onto an imaging sensor. Coupling an optical model with deep learning, our system recovers the 3D holographic phase and class identity for six waterborne particle types, exemplified by microplastics, from a single snapshot. This enables all-in-one reconstruction of both geometry and identity. Both supervised and unsupervised pipelines outperform hologram-only baselines, achieving a macro F1 of 98.10%. The F1 score combines precision and recall, offering a comprehensive evaluation of classification performance. With the unsupervised feature-extraction route, our method improves F1 by 15%, greatly reducing reliance on labeled data. The approach enables long-duration, real-time monitoring in dynamic aquatic environments without sample pretreatment or labeling, and it extends readily from transmission spectroscopy to other spectral modalities.