9 June 2020 in IEEE Internet of Things Journal
A key challenge for autonomous driving is to process a massive amount of sensor data and make safe and reliable decisions in real-time. However, autonomous vehicles often have insufficient on-board resources to provide the required computation capacity. To address this problem, this paper advocates a novel approach to offload computation-intensive autonomous driving services to road-side units and cloud for swift executions. Our approach combines an Integer Linear Programming (ILP) formulation for offline optimization of the scheduling strategy and a fast heuristics algorithm for online adaptation. We verify our technique with both synthetic task graphs and real-world deployment. Experimental results show that our approach can improve system performance effectively.
19 January 2022 in IEEE Transactions on Instrumentation and Measurement
Depth-map estimation reflects the geometry of the visible surface in the environment directly and plays an important role in perception and decision for intelligent robots. However, sparse LiDAR only provides low-resolution depth information, which is a huge challenge for accurate sensing algorithms. To address this problem, this article proposes a novel fusion framework to generate dense depth-map based on event camera and sparse LiDAR. The approach uses the geometric information provided by the point cloud as prior knowledge and clusters point cloud data by an improved density clustering algorithm. Combined with the 3-D surface model of each cluster, the approach can provide 3-D reconstructions of the coordinate points of events and further obtain the dense-depth map by depth expansion and hole filling. Finally, we deploy our approach in MVSEC datasets and real-world applications. Experimental results show that, compared with other approaches, our approach can obtain more accurate depth information.
26 July 2021 in IEEE Transactions on Instrumentation and Measurement
Point clouds from 3-D light detection and ranging (LiDAR) are useful for roadside units (RSU) applications in intelligent transportation system (ITS). High-density LiDAR products are restricted by high cost while the low-density and cheap ones are usually insufficient to perceive. Event-based cameras react to the changes in light intensity and output dense event streams consisting of triggered pixels. Unfortunately, there currently lacks depth information for event cameras. In order to address these problems, this article presents an approach to enhance sparse 3-D LiDAR point clouds with event pixels from an event-based camera. In our approach, the depth values of event pixels are estimated based on the distribution models which they belong to. The distribution models of event pixels are determined by the spatial information of the neighboring LiDAR points in a structural manner which we called the physical structure. To verify our approach, we conduct several real-world experiments about RSU applications in ITS. Results demonstrate that our approach can effectively improve 3-D point clouds density. The average accuracy of 3-D and 2-D vehicle detection increase by a factor of 14.6 and 8.8, respectively.
13 April 2017 in PLOS ONE
Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D conversion process. Due to the absence of knowledge about the 3D structure of a scene and its corresponding texture, DIBR in the 2D to 3D conversion process, inevitably leads to holes in the resulting 3D image as a result of newly-exposed areas. In this paper, we proposed a structure-aided depth map preprocessing framework in the transformed domain, which is inspired by recently proposed domain transform for its low complexity and high efficiency. Firstly, our framework integrates hybrid constraints including scene structure, edge consistency and visual saliency information in the transformed domain to improve the performance of depth map preprocess in an implicit way. Then, adaptive smooth localization is cooperated and realized in the proposed framework to further reduce over-smoothness and enhance optimization in the non-hole regions. Different from the other similar methods, the proposed method can simultaneously achieve the effects of hole filling, edge correction and local smoothing for typical depth maps in a united framework. Thanks to these advantages, it can yield visually satisfactory results with less computational complexity for high quality 2D to 3D conversion. Numerical experimental results demonstrate the excellent performances of the proposed method.
7 August 2018 in IEEE Transactions on Industrial Informatics
Scene flow is an essential part of stereo-based perception system for autonomous driving and mobile robotics. As in most of these platforms, the computing resource is limited but the computing requirement is high, embedded and parallelized algorithms are of vital importance for real-time tasks. This paper develops a cross-platform embedded scene flow algorithm by using an OpenCL (Open Computing Language) programming. Meanwhile, we propose a method to achieve a good performance by using a novel coarse-grained software pipeline for the embedded stream application. Experimental results show that the proposed algorithm can boost the average processing speed to 50 fps for different commercial-off-the-shelf (COTS) hardware, including desktop GPUs, FPGAs and mobile phone platforms. For certain GPUs the peak frame rates can also reach 1000 fps. By comparing the efficiency among the serial platform, we illustrate that with the help of OpenCL programming, COTS platforms can provide enough computing resources for the stereobased perception algorithm.
28 August 2024 in IEEE Transactions on Biomedical Circuits and Systems
Wearable devices with continuous monitoring capabilities are critical for the daily detection of epileptic seizures, as they provide users with accurate and comprehensible analytical results. However, current AI classifiers rely on a two-stage recognition process for continuous monitoring, which only reduces operation time but remains challenged by the high cost of additional hardware. To address this problem, this article proposes a novel fusion architecture for AI processors, which enables event-triggered cross-paradigm integration and computation. Our method introduces a distributed-aggregated classification architecture (D-ACA) that facilitates the reuse of hardware resources across two-stage recognition, thereby obviating the need for standby hardware and enhancing energy efficiency. Integrating a non-encoding biomedical circuit method based on spiking neural networks (SNNs), the architecture eliminates encoded neurons at the hardware level, significantly optimizing energy consumption and hardware resource utilization. Additionally, we develop a configurable and highly flexible control method that supports various neuron modules, enabling continuous detection of epileptic seizures and activating high-precision recognition upon event detection. Finally, we implement the design on the Xilinx ZCU 102 FPGA board, where the AI processor achieves a high classification accuracy of 98.1% while consuming extremely low classification energy (3.73 μJ per classification).