Optical flow is a crucial component of the feature space for early visual processing of dynamic scenes especially in new applications such as self-driving vehicles, drones and autonomous robots. The dynamic vision sensors are well suited for such applications because of their asynchronous, sparse and temporally precise representation of the visual dynamics. Many algorithms proposed for computing visual flow for these sensors suffer from the aperture problem as the direction of the estimated flow is governed by the curvature of the object rather than the true motion direction. Some methods that do overcome this problem by temporal windowing under-utilize the true precise temporal nature of the dynamic sensors. In this paper, we propose a novel multi-scale plane fitting based visual flow algorithm that is robust to the aperture problem and also computationally fast and efficient. Our algorithm performs well in many scenarios ranging from fixed camera recording simple geometric shapes to real world scenarios such as camera mounted on a moving car and can successfully perform event-by-event motion estimation of objects in the scene to allow for predictions of up to 500 ms i.e. equivalent to 10 to 25 frames with traditional cameras.

This paper studies the suitability of neuromorphic event-based vision cameras for spaceflight and the effects of neutron radiation on their performance. Neuromorphic event-based vision cameras are novel sensors that implement asynchronous, clockless data acquisition, providing information about the change in illuminance ≥ 120dB with sub-millisecond temporal precision. These sensors have huge potential for space applications as they provide an extremely sparse representation of visual dynamics while removing redundant information, thereby conforming to low-resource requirements. An event-based sensor was irradiated under wide-spectrum neutrons at Los Alamos Neutron Science Center and its effects were classified. Radiation-induced damage of the sensor under wide-spectrum neutrons was tested, as was the radiative effect on the signal-to-noise ratio of the output at different angles of incidence from the beam source. We found that the sensor had very fast recovery during radiation, showing high correlation of noise event bursts with respect to source macro-pulses. No statistically significant differences were observed between the number of events induced at different angles of incidence but significant differences were found in the spatial structure of noise events at different angles. The results show that event-based cameras are capable of functioning in a space-like, radiative environment with a signal-to-noise ratio of 3.355. They also show that radiation-induced noise does not affect event-level computation. Finally, we introduce the Event-based Radiation-Induced Noise Simulation Environment (Event-RINSE), a simulation environment based on the noise-modelling we conducted and capable of injecting the effects of radiation-induced noise from the collected data to any stream of events in order to ensure that developed code can operate in a radiative environment. To the best of our knowledge, this is the first time such analysis of neutron-induced noise has been performed on a neuromorphic vision sensor, and this study shows the advantage of using such sensors for space applications.

Event-driven vision sensors have the potential to support a new generation of efficient and robust robots. This requires the development of a new computational framework that exploits not only the spatial information, like in the traditional frame-based approach, but also the temporal content of the sensory data. We propose a method for unsupervised learning of filters for the processing of the visual signal from event-driven sensors. This method exploits the temporal coincidence of events generated by each object in a spatial location of the visual field. The approach is based on a modification of Spike Timing Dependent Plasticity that takes into account the specific implementation on the robot and the characteristics of the used sensor. It gives rise to oriented spatial filters that are very similar to the receptive fields observed in the primary visual cortex and traditionally used in bio-inspired hierarchical structures for object recognition, as well as to novel curved spatial structures. Using mutual information measure we provide a quantitative evidence that such curved spatial filters provide more information than equivalent oriented Gabor filters and can be an important aspect for object recognition in robotic applications.

Event-based vision sensors produce asynchronous event streams with high temporal resolution based on changes in the visual scene. The properties of these sensors allow for accurate and fast calculation of optical flow as events are generated. Existing solutions for calculating optical flow from event data either fail to capture the true direction of motion due to the aperture problem, do not use the high temporal resolution of the sensor, or are too computationally expensive to be run in real time on embedded platforms. In this research, we first present a faster version of our previous algorithm, ARMS (Aperture Robust Multi-Scale flow). The new optimized software version (fARMS) significantly improves throughput on a traditional CPU. Further, we present hARMS, a hardware realization of the fARMS algorithm allowing for real-time computation of true flow on low-power, embedded platforms. The proposed hARMS architecture targets hybrid system-on-chip devices and was designed to maximize configurability and throughput. The hardware architecture and fARMS algorithm were developed with asynchronous neuromorphic processing in mind, abandoning the common use of an event frame and instead operating using only a small history of relevant events, allowing latency to scale independently of the sensor resolution. This change in processing paradigm improved the estimation of flow directions by up to 73% compared to the existing method and yielded a demonstrated hARMS throughput of up to 1.21 Mevent/s on the benchmark configuration selected. This throughput enables real-time performance and makes it the fastest known realization of aperture-robust, event-based optical flow to date.

As the use of neuromorphic, event-based vision sensors expands, the need for compression of their output streams has increased. While their operational principle ensures event streams are spatially sparse, the high-temporal resolution of the sensors can result in high-data rates from the sensor depending on scene dynamics. For systems operating in communication-bandwidth-constrained and power-constrained environments, it is essential to compress these streams before transmitting them to a remote receiver. Therefore, we introduce a flow-based method for the real-time asynchronous compression of event streams as they are generated. This method leverages real-time optical flow estimates to predict future events without needing to transmit them, therefore, drastically reducing the amount of data transmitted. The flow-based compression introduced is evaluated using a variety of methods, including spatiotemporal distance between event streams. The introduced method itself is shown to achieve an average compression ratio (CR) of 2.70 on a variety of event-camera data sets with the evaluation configuration used. That compression is achieved with a median temporal error of 0.31 ms and an average spatiotemporal event-stream distance of 4.72. When combined with Lempel-Ziv–Markov chain algorithm compression for non-real-time applications, our method can achieve state-of-the-art average CRs ranging from 9.29 to 13.01. Additionally, we demonstrate that the proposed prediction algorithm is capable of performing real time, low-latency event prediction.