Safety is the most important requirement for autonomous vehicles; hence, the ultimate challenge of designing an edge computing ecosystem for autonomous vehicles is to deliver enough computing power, redundancy, and security so as to guarantee the safety of autonomous vehicles. Specifically, autonomous driving systems are extremely complex; they tightly integrate many technologies, including sensing, localization, perception, decision making, as well as the smooth interactions with cloud platforms for high-definition (HD) map generation and data storage. These complexities impose numerous challenges for the design of autonomous driving edge computing systems. First, edge computing systems for autonomous driving need to process an enormous amount of data in real time, and often the incoming data from different sensors are highly heterogeneous. Since autonomous driving edge computing systems are mobile, they often have very strict energy consumption restrictions. Thus, it is imperative to deliver sufficient computing power with reasonable energy consumption, to guarantee the safety of autonomous vehicles, even at high speed. Second, in addition to the edge system design, vehicle-to-everything (V2X) provides redundancy for autonomous driving workloads and alleviates stringent performance and energy constraints on the edge side. With V2X, more research is required to define how vehicles cooperate with each other and the infrastructure. Last, safety cannot be guaranteed when security is compromised. Thus, protecting autonomous driving edge computing systems against attacks at different layers of the sensing and computing stack is of paramount concern. In this paper, we review state-of-the-art approaches in these areas as well as explore potential solutions to address these challenges.

To simultaneously enable multiple autonomous driving services on affordable embedded systems, we designed and implemented LoPECS, a Low-Power Edge Computing System for real-time autonomous robots and vehicles services. The contributions of this paper are three-fold: first, we developed a Heterogeneity-Aware Runtime Layer to fully utilize vehicle’s heterogeneous computing resources to fulfill the real-time requirement of autonomous driving applications; second, we developed a vehicle-edge Coordinator to dynamically offload vehicle tasks to edge cloudlet to further optimize user experience in the way of prolonged battery life; third, we successfully integrated these components into LoPECS system and implemented it on Nvidia Jetson TX1. To the best of our knowledge, this is the first complete edge computing system in a production autonomous vehicle. Our implementation on Nvidia Jetson demonstrated that it could successfully support multiple autonomous driving services with only 11 W of power consumption, and hence proves the effectiveness of the proposed LoPECS system.

Bundle adjustment (BA) is a fundamental optimization technique used in many crucial applications, including 3D scene reconstruction, robotic localization, camera calibration, autonomous driving, street view map generation, and even space exploration etc. Essentially, BA is a joint non-linear optimization problem, and one which can consume a significant amount of time and power, especially for large optimization problems. Previous approaches of optimizing BA performance heavily rely on parallel processing or distributed computing, which trade higher power consumption for higher performance. In this paper we propose π-BA, the first hardware-software co-designed BA hardware accelerator that exploits custom hardware to simultaneously achieve higher performance and power efficiency. Specifically, based on our key observation that not all 3D points appear on all images in a BA problem, we designed a Co-Observation Optimization technique to accelerate BA operations with optimized usage of memory and computation resources. In addition, we developed a hardware-friendly differentiation method, which combines the analytic and forward automatic differentiation to calculate derivatives of projection function in the BA problem. We have implemented the proposed design on an embedded FPGA SoC, and experimental results confirm that π-BA outperforms the existing software implementations in terms of performance and power consumption.

In recent years, we have observed a clear trend in the rapid rise of autonomous vehicles and robotics. One of the core technologies enabling these applications, Simultaneous Localization And Mapping (SLAM), imposes two main challenges: first, these workloads are computationally intensive and they often have real-time requirements; second, these workloads run on battery-powered mobile devices with limited energy budget. Hence, performance should be improved while simultaneously reducing energy consumption, two rather contradicting goals by conventional wisdom. Previous attempts to optimize SLAM performance and energy efficiency usually involve optimizing one function and fail to approach the problem systematically. In this paper, we first study the characteristics of visual inertial SLAM workloads on existing heterogeneous SoCs. Then based on the initial findings, we propose π-SoC, a heterogeneous SoC design that systematically optimize the IO interface, the memory hierarchy, as well as the the hardware accelerator. We implemented this system on a Xilinx Zynq UltraScale MPSoC and was able to deliver over 60 FPS performance with average power less than 5 W.

Despite extensive efforts, existing approaches to design accelerators for optimization-based robotic applications have limitations. Some approaches focus on accelerating general matrix operations, but they fail to fully exploit the specific sparse structure commonly found in many robotic algorithms. On the other hand, certain methods require manual design of dedicated accelerators, resulting in inefficiencies and significant non-recurring engineering (NRE) costs. The Orianna framework represents a significant advancement as it overcomes the limitations of existing approaches and provides a solution to generate accelerators for robotic applications that integrate multiple optimization-based algorithms. The framework leverages a common abstraction factor graph to overcome these challenges. Application designers can easily define their objectives and constraints using the Orianna software framework. The Orianna compiler then translates the high-level code into basic matrix operations and executes them in an out-of-order manner on the generated accelerator. We demonstrate its effectiveness by implementing a real FPGA-based prototype. Orianna accelerators achieve a 6.5× performance improvement and a 15.1× energy reduction compared to an advanced Intel CPU. Furthermore, Orianna achieves comparable performance to state-of-the-art dedicated accelerators while utilizing only 36.1% of hardware resources.

Time synchronization is a critical task in robotic computing such as autonomous driving. In the past few years, as we developed advanced robotic applications, our synchronization system has evolved as well. In this paper, we first introduce the time synchronization problem and explain the challenges of time synchronization, especially in robotic workloads. Summarizing these challenges, we then present a general hardware synchronization system for robotic computing, which delivers high synchronization accuracy while maintaining low energy and resource consumption. The proposed hardware synchronization system is a key building block in our future robotic products.