Event camera emerges as a new type of neuromorphic sensor, records scenes in an asynchronous paradigm which provides high temporal resolution and high dynamic range. The huge volume of event-based data brings computational burden to data transmission and real-time interaction in intelliSense applications. Compared with general encoding method and current event-based data compression method Spike Coding, we successively propose two event-based data compression methods by analyzing the statistical features of the event characteristic parameters (x, y, t, f). Based on the data features of event parameters, we design the " Characteristic Parameter Jointed Coding (CPJC)" method with the idea of fixed-length coding, which reduces redundancy of delimiters between modules. As for the limitation of CPJC method in pixel scale, we further propose the "ASCII Coding based on Bit Operation (ACBO) " method based on the data storage principle in computer memory, reducing the redundancy caused by data storage. We introduce compression coefficient K as the metrics to evaluate the compression performance on real-world dataset. And the metrics K of the proposed two methods is improved by 17.93% and 14.92%, respectively, compared with the current event-based compression method. The proposed ACBO method also eliminate the limitation in data pixel scale, and have certain adoption for event noise and event dispersion in the scene.
Deep learning has rapidly advanced amidst the proliferation of large models, leading to challenges in computational resources and power consumption. Optical neural networks (ONNs) offer a solution by shifting computation to optics, thereby leveraging the benefits of low power consumption, low latency, and high parallelism. The current training paradigm for ONNs primarily relies on backpropagation (BP). However, the reliance is incompatible with potential unknown processes within the system, which necessitates detailed knowledge and precise mathematical modeling of the optical process. In this paper, we present a pre-sensor multilayer ONN with nonlinear activation, utilizing a forward-forward algorithm to directly train both optical and digital parameters, which replaces the traditional backward pass with an additional forward pass. Our proposed nonlinear optical system demonstrates significant improvements in image classification accuracy, achieving a maximum enhancement of 9.0%. It also validates the efficacy of training parameters in the presence of unknown nonlinear components in the optical system. The proposed training method addresses the limitations of BP, paving the way for applications with a broader range of physical transformations in ONNs.
Non-line-of-sight (NLOS) imaging with intelligent sensors emerges as a novel technique in imaging and sensing occluded objects around corners. With the innovation of bio-inspired neuromorphic sensors, the applications of novel sensors in unconventional imaging tasks like NLOS imaging have shown promising prospects in intelligent perception, encompassing autonomous driving, medical endoscopy and other sensing scenarios. However, the most challenging point of sensors application in computational imaging is the inverse problem established between sensors acquisition and reconstructions. Traditional physical retrieval methods with certain sensors applications usually result in poor reconstruction due to the highly ill-posedness, particularly in moving object imaging. Thanks to the development of neural networks, data-driven methods have greatly improved its accuracy, however, heavy reliance on data volume has put great pressure on data collection and dataset fabrication. To the best of our knowledge, we firstly propose a sensor-dominated restoration prototype termed “event enhanced passive NLOS imaging prototype for moving objects with physical embedding” (EPNP), which illustrated the application of dynamic vision sensors in NLOS imaging. EPNP induces an event camera for feature extraction of dynamic diffusion spot and leverages simulation dataset to pre-train the physical embedded model before fine-tuning with limited real-shot data. The proposed EPNP prototype is verified by simulation and real-world experiments, while the comparisons of data paradigms also validate the superiority of event-based sensor applications in passive NLOS imaging for moving objects and perspectives in advanced imaging techniques.
Fully connected computing is a crucial computational function in neural networks. However, in the implementation of optical neural networks, there exist issues such as the absence of fully connected layers under incoherent light conditions and high overhead caused by reliance on electronic layers. To address these problems, this paper proposes an incoherent optical fully connected computing method based on light intensity modulation. A fully connected computing system is constructed using a microlens array (MLA), an amplitude-type liquid crystal spatial light modulator (SLM), and an image sensor. By utilizing the MLA to replicate the target scene into a sub-image array, combined with pixel-wise weight modulation via the SLM and data acquisition by the image sensor, a multi-input multi-output fully connected computing architecture is realized. The design principles for system parameters are derived theoretically, and experiments verify the designed fully connected computing function. A 6-bit linear modulation is achieved through SLM calibration. Finally, an optical neural network system is built, and a classification accuracy of 93.5% on the MNIST dataset is achieved. As an important component of optical neural networks research, this system will be applied to the implementation of all-optical neural networks and holds significant application prospects in the era of artificial intelligence.
Optical computing for artificial neural network implementation emerges as an important perspective for AI-assisted vision perception. However, the suitable utilization of optical devices and efficient optical computing models remains a challenge in the establishment of the optical network. In this paper, we leverage the grating diffraction effect of spatial light modulators for multiplexing and achieve an optical feature extraction computational prototype in image perception. Simulations are conducted to verify the established physical model, and real-shot experiments on the MNIST dataset and the UC Merced dataset validate the feasibility of the established scheme. The proposed optical multiplexing method with a lightweight backend processor achieves an accuracy of 93% in 5-class MNIST classification and 82% in UC Merced remote sensing scene classification. The proposed method achieves a compression ratio of 6.27 via feature encoding. It shows promising potential in utilizing the inherent physical properties of light propagation and information multiplexing in optical neural networks.
An event camera is a neuromorphic vision sensor with a high dynamic range (HDR), high temporal resolution (HTR), low latency, and low power consumption. A polarimeter is an instrument for measuring the state of polarization of light. Currently, most imaging polarimeters are limited in dynamic range and frame rate when used with frame-based cameras. We can establish an event-based imaging polarimeter using the principle of the event camera to obtain HDR and HTR polarized event streams for processing polarization information. However, because of the short history and high cost of event cameras, developing an event-based imaging polarimeter requires substantial resources. We propose an event-based imaging polarimeter simulation method with a single division of focal plane image based on the existing research on event simulation. This method can easily convert existing data into a polarized event stream. It is beneficial to lower the requirement of processing polarized event streams and to create large datasets for deep learning.