4 April 2022 in IEEE Transactions on Image Processing
Event cameras, offering extremely high temporal resolution and high dynamic range, have brought a new perspective to addressing common object detection challenges (e.g., motion blur and low light). However, how to learn a better spatio-temporal representation and exploit rich temporal cues from asynchronous events for object detection still remains an open issue. To address this problem, we propose a novel asynchronous spatio-temporal memory network (ASTMNet) that directly consumes asynchronous events instead of event images prior to processing, which can well detect objects in a continuous manner. Technically, ASTMNet learns an asynchronous attention embedding from the continuous event stream by adopting an adaptive temporal sampling strategy and a temporal attention convolutional module. Besides, a spatio-temporal memory module is designed to exploit rich temporal cues via a lightweight yet efficient inter-weaved recurrent-convolutional architecture. Empirically, it shows that our approach outperforms the state-of-the-art methods using the feed-forward frame-based detectors on three datasets by a large margin (i.e., 7.6% in the KITTI Simulated Dataset, 10.8% in the Gen1 Automotive Dataset, and 10.5% in the 1Mpx Detection Dataset). The results demonstrate that event cameras can perform robust object detection even in cases where conventional cameras fail, e.g., fast motion and challenging light conditions.
23 November 2021 in IEEE Transactions on Circuits and Systems for Video Technology
Spike camera is a new type of bio-inspired vision sensor, each pixel of which perceives the brightness of the scene independently, and finally outputs 3-dimensional spatiotemporal spike streams. To bridge the spike camera and traditional frame-based vision, there is some works to reconstruct spike streams into regular images. However, the low spatial resolution (400×250) of the spike camera limits the quality of the reconstructed images. Thus, it is meaningful to explore a super-resolution reconstruction for spike streams. In this paper, we propose an end-to-end network to reconstruct high-resolution images from low-resolution spike streams. To utilize more spatiotemporal features of spike streams, our network adopts a multi-level features learning mechanism, including intra-stream feature extraction by spike encoder, inter-stream dependencies extraction based on optical flow module, and joint features learning via spike-based iterative projection. Experimental results demonstrate that our network is superior to the combination of state-of-the-art intensity image reconstruction methods and super-resolution networks on simulated and real datasets.
22 May 2026 in IEEE Transactions on Circuits and Systems for Video Technology
Event cameras offer unique advantages such as high temporal resolution and low power consumption. However, under slow-motion or low-light scenarios, the resulting event streams are often sparse, limiting the performance of event-based vision algorithms. In this paper, we propose a novel temporal upsampling framework that enhances the temporal resolution of sparse events by estimating high-frame-rate brightness changes. Our method first reconstructs long-term brightness changes from sparse input events using a neural network, then optimizes short-term high-frame-rate brightness variations through a spatiotemporal smoothness and sparsity-constrained model. Following the generation principle of event cameras, new events are synthesized when estimated brightness changes exceed the firing threshold. To evaluate the effectiveness of our method, we construct a temporal upsampling dataset and introduce two event-based metrics to directly assess the quality of upsampled events. Extensive experiments demonstrate that our approach can generate denser and more informative events, significantly enhancing the performance of downstream tasks, including image reconstruction, optical flow estimation, and object detection in sparse conditions.