1 April 2021 in APL Photonics
An opto-electronic neural network is designed for video object detection from a long-exposure blurred image. This network combines an optical encoder, convolutional neural network decoder, and object detection module, which are jointly optimized end-to-end. The joint loss is adopted for updating the network according to the physical constraints of hardware via back-propagation. A high-speed refreshed spatial light modulator is used as the encoder part of the network to generate coded sub-images, and then, a single blurred image is obtained after a common camera. The rest of the network is used for video object detection. Both simulations and experiments demonstrate that our framework can successfully retrieve object labels and bounding boxes at different moments in the long exposure. To the best of our knowledge, this is the first work investigating video object detection from a single motion-degraded image.
7 October 2022 in Optics Express
Snapshot compressive imaging (SCI) encodes high-speed scene video into a snapshot measurement and then computationally makes reconstructions, allowing for efficient high-dimensional data acquisition. Numerous algorithms, ranging from regularization-based optimization and deep learning, are being investigated to improve reconstruction quality, but they are still limited by the ill-posed and information-deficient nature of the standard SCI paradigm. To overcome these drawbacks, we propose a new key frames assisted hybrid encoding paradigm for compressive video sensing, termed KH-CVS, that alternatively captures short-exposure key frames without coding and long-exposure encoded compressive frames to jointly reconstruct high-quality video. With the use of optical flow and spatial warping, a deep convolutional neural network framework is constructed to integrate the benefits of these two types of frames. Extensive experiments on both simulations and real data from the prototype we developed verify the superiority of the proposed method.
15 September 2020 in IEEE Photonics Journal
Ghost imaging is a fascinating framework which constructs the image of an object by correlating measurements between received beams and reference beams, none of which carries the structure information of the object independently. Recently, by taking into account spacetime duality in optics, computational temporal ghost imaging has attracted attention. Here, we propose a novel Fourier temporal ghost imaging (FTGI) scheme to achieve single-shot nonreproducible temporal signals. By sinusoidal coded modulation, ghost images are obtained and recovered by applying Fourier transformation. For demonstration, non-repeating events are detected with single-shot exposure architecture. It is shown in experimental results that the peak signal-to-noise ratio (PSNR) of FTGI is significantly better (13dB increase) than traditional temporal ghost imaging in the same condition. In addition, by using the obvious physical meaning of the Fourier spectrum, we show some potential applications of FTGI, such as frequency division multiplexing demodulation in the visible light communications.
1 November 2022 in APL Photonics
The framework provides a novel pipeline for action recognition. The action recognition task classifies the action label of the scene. High-speed cameras are commonly used to generate high frame-rate videos for capturing sufficient motion information. However, the data volume would be the bottleneck of the system. With the insight that the discrete cosine transform (DCT) of video signals reveals the motion information remarkably, instead of obtaining video data as with traditional cameras, the proposed method directly captures a DCT spectrum of video in a single shot through optical pixel-wise encoding. Considering that video signals are sparsely distributed in the DCT domain, a learning-based frequency selector is designed for pruning the trivial frequency channels of the spectrum. An opto-electronic neural network is designed for action recognition from a single coded spectrum. The optical encoder generates the DCT spectrum, and the rest of the network jointly optimizes the frequency selector and classification model simultaneously. Compared to conventional video-based action recognition methods, the proposed method achieves higher accuracy with less data, less communication bandwidth, and less computational burden. Both simulations and experiments demonstrate that the proposed method has superior action recognition performance. To the best of our knowledge, this is the first work investigating action recognition in the DCT domain.