1 December 2020 in Neural Networks
3 March 2023 in IEEE Transactions on Image Processing
In the past years, attention-based Transformers have swept across the field of computer vision, starting a new stage of backbones in semantic segmentation. Nevertheless, semantic segmentation under poor light conditions remains an open problem. Moreover, most papers about semantic segmentation work on images produced by commodity frame-based cameras with a limited framerate, hindering their deployment to auto-driving systems that require instant perception and response at milliseconds. An event camera is a new sensor that generates event data at microseconds and can work in poor light conditions with a high dynamic range. It looks promising to leverage event cameras to enable perception where commodity cameras are incompetent, but algorithms for event data are far from mature. Pioneering researchers stack event data as frames so that event-based segmentation is converted to framebased segmentation, but characteristics of event data are not explored. Noticing that event data naturally highlight moving objects, we propose a posterior attention module that adjusts the standard attention by the prior knowledge provided by event data. The posterior attention module can be readily plugged into many segmentation backbones. Plugging the posterior attention module into a recently proposed SegFormer network, we get EvSegFormer (the event-based version of SegFormer) with state-of-the-art performance in two datasets (MVSEC and DDD-17) collected for event-based segmentation. Code is available at https://github.com/zexiJia/EvSegFormer to facilitate research on event-based vision.
16 August 2024 in Nature Computational Science
30 August 2021 in IEEE Transactions on Circuits and Systems II: Express Briefs
This brief presents a novel spatio-temporal domain adaptation (DA) method that is unsupervised and event-driven for dynamic vision sensor (DVS) gesture recognition. This method realizes the transfer between the source domain and the target domain data in both the spatial and temporal dimension without the need of target domain data labels. Specifically, it consists of a deep spiking neural network (SNN)-based feature extractor, a label predictor and a domain discriminator. A time-space gradient reversal layer is responsible for building a spatio-temporal bridge between the domain discriminator and the feature extractor, which is essential to the alignment of the source domain spike features with the target ones and to achieve domain adaptation on both space and time dimensions. To demonstrate the effectiveness of our method, we adapted DVS hand gesture data from one temporal resolution to another and from original data to denoised data. Our method can provide up to 10.39% improvement in accuracy. Its accuracy improvement is more stable comparing with RNN-based and LSTM-based methods in this DA framework especially when the two domains have partial similarity in DVS data.
21 April 2024 in Neural Networks