In modern cyber-physical systems, the integration of AI into vision pipelines is now a standard practice for applications ranging from autonomous vehicles to mobile devices. Traditional AI integration often relies on cloud-based processing, which faces challenges such as data access bottlenecks, increased latency, and high power consumption. This article reviews embedded AI vision systems, examining the diverse landscape of near-sensor and in-sensor processing architectures that incorporate convolutional neural networks. We begin with a comprehensive analysis of the critical characteristics and metrics that define the performance of AI-integrated vision systems. These include sensor resolution, frame rate, data bandwidth, computational throughput, latency, power efficiency, and overall system scalability. Understanding these metrics provides a foundation for evaluating how different embedded processing architectures impact the entire vision pipeline, from image capture to AI inference. Our analysis delves into near-sensor systems that leverage dedicated hardware accelerators and commercially available components to efficiently process data close to their source, minimizing data transfer overhead and latency. These systems offer a balance between flexibility and performance, allowing for real-time processing in constrained environments. In addition, we explore in-sensor processing solutions that integrate computational capabilities directly into the sensor. This approach addresses the rigorous demand constraints of embedded applications by significantly reducing data movement and power consumption while also enabling in-sensor feature extraction, pre-processing, and CNN inference. By comparing these approaches, we identify trade-offs related to flexibility, power consumption, and computational performance. Ultimately, this article provides insights into the evolving landscape of embedded AI vision systems and suggests new research directions for the development of next-generation machine vision systems.
7 February 2022 in IEEE Transactions on Computational Imaging
Single photon avalanche diodes (SPADs) combined with high-frequency time-to-digital converters (TDCs) enable the estimation of photon Time-of-Flight (ToF) for active 3D-depth imaging. Nevertheless, SPAD sensors still face hardware limitations due to a complex pixel readout design and a large amount of data collected by way of pixel-wise histograms. The intrinsic high background illumination (BI) also remains a challenging issue for the related depth reconstruction algorithms. Using a physically-relevant SPAD sensor model, this work tackles these issues by implementing a pixel-wise ToF histogram compressive sensing (CS) with a specific deep generative model based reconstruction. It demonstrates a possible reduction of hardware design constraints while reaching a depth inference root mean square error below 16 centimeters regardless of BI ${(50 \times 1050 \ W/m^{2})}$ and distance (20m), at a compression ratio (CR) of 10% (32 CS measurements). In addition, this paper introduces a novel multimodal reconstruction from SPAD data, enabling joint depth and luminance estimations. Indeed, since ToF histogram raw data gathers multiple physical scene characteristics, we propose a two-part deep generative model (DGM) capable of inferring Super-Resolved depth maps and normalized luminance images, independently from the average scene BI. Our key contributions related to the DGM topology design are the introduction of proper normalization layers with a learned pile-up effect compensation, multidimensional-multiscale filtering and the concatenation of Softmax-ReLU activation functions to capture both peak-position and relative amplitude features. Numerically, depth and luminance maps reconstructions of natural scenes respectively reach more than 30dB and 25dB PSNRs for any CR higher than 2.5%.
1 November 1998 in Annals of the New York Academy of Sciences
This work presents a compact CMOS Image Sensor (CIS) architecture enabling embedded object recognition facilitated by a dedicated end-of-column Compressive Sensing (CS), reducing on-chip memory needs. Our sensing scheme is based on a combination of random modulations and permutations leading to an implementation with very limited hardware impacts. It is designed to meet both theoretical (i.e., stable embedding, measurements incoherence) and practical requirements (i.e., silicon footprint, power consumption). The only additional hardware compared to a standard CIS architecture using first order incremental Sigma-Delta (Σ Δ) Analog to Digital Converter (ADC) are a pseudo-random data mixing circuit, an in-Σ Δ ± 1 modulator and a small Digital Signal Processor (DSP). On the algorithmic side, three variants are presented to perform the inference on compressed measurements with a tunable complexity (i.e., one-vs.-all SVM, hierarchical SVM and small ANN with 1-D max-pooling). An object recognition accuracy of ~eq98.8% is reached on the COIL database (COIL, 100 classes) using our dedicated Neural Network classifier. We stress that the signal-independent dimensionality reduction performed by our dedicated CS scheme (1/480 in 480 x 640 VGA resolution case) allows to dramatically reduce memory requirements mainly related to the remotely learned coefficients used for the inference stage.
1 January 1997 in Intellectica. Revue de l'Association pour la Recherche Cognitive
Representations of olfactory space : from receptors to perception. Non expert subjects have acquired the ability to describe precisely their visual or auditory perceptions but their description of smells are usually short. In some case they name the odour source (lemon) but they often limit their comments to the pleasant/unpleasant feelings evoked by their olfactory sensations. This makes difficult to obtain a good representation of the human perceptual olfactory space. In order to overpass this difficulty, we asked subjects to put odorant samples in groups according to their "family resemblance". One of the major results of the study was that every subject proposed his own arrangement. From 40 subjects faced to this grouping task of 20 odorants, it was possible to extract the most common clusters used by subjects. The same series of odorant chemicals can be regarded with other points of view or at other levels of coding in the olfactory system. Chemists can use a number of different physico-chemical parameters to describe the odorant molecules and determine those playing a role in the olfactory space. Physiologists obtained consistent data on structure and function of the nervous system which support the olfactory information processing from studies in the vertebrates. For instance, we have shown images of the activities evoked by odours in the olfactory mucosa of frogs. Thus, we can compare these different representations of odours despite the inherent limitations of such a generalization.