The last five years marked a surge in interest for and use of smart robots, which operate in dynamic and unstructured environments and might interact with humans. We posit that well-validated computer simulation can provide a virtual proving ground that in many cases is instrumental in understanding safely, faster, at lower costs, and more thoroughly how the robots of the future should be designed and controlled for safe operation and improved performance. Against this backdrop, we discuss how simulation can help in robotics, barriers that currently prevent its broad adoption, and potential steps that can eliminate some of these barriers. The points and recommendations made concern the following simulation-in-robotics aspects: simulation of the dynamics of the robot; simulation of the virtual world; simulation of the sensing of this virtual world; simulation of the interaction between the human and the robot; and, in less depth, simulation of the communication between robots. This Perspectives contribution summarizes the points of view that coalesced during a 2018 National Science Foundation/Department of Defense/National Institute for Standards and Technology workshop dedicated to the topic at hand. The meeting brought together participants from a range of organizations, disciplines, and application fields, with expertise at the intersection of robotics, machine learning, and physics-based simulation.

Simulation can be an important tool in evaluating autonomous vehicle performance. In a virtual proving ground, challenging and highly unlikely scenarios can be created and novel control algorithms can be tested with no safety concerns and relatively little cost. A critical component of any virtual proving ground is sensor simulation; i.e., the ability to produce realistic sensor information inside the simulated environment. This paper overviews several models and methods for generating, through simulation, GPS, IMU, and lidar data. While the current models can account for some sensor noise, significant work remains to be done in order to produce simulated data that is realistic and can be used to confidently test and evaluate autonomous agents in virtual proving grounds.

Computer simulation can be a useful tool when designing robots expected to operate independently in unstructured environments. In this context, one needs to simulate the dynamics of the robot’s mechanical system, the environment in which the robot operates, and the sensors which facilitate the robot’s perception of the environment. Herein, we focus on the sensing simulation task by presenting a virtual sensing framework built alongside an open-source, multi-physics simulation platform called Chrono. This framework supports camera, lidar, GPS, and IMU simulation. We discuss their modeling as well as the noise and distortion implemented to increase the realism of the synthetic sensor data. We close with two examples that show the sensing simulation framework at work: one pertains to a reduced scale autonomous vehicle and the second is related to a vehicle driven in a digital replica of a Madison neighborhood.

Simulation is increasingly important in the development and testing of robots and autonomous vehicles as it opens the door for candidate navigation, perception, and sensor fusion algorithms to be expeditiously probed in complex and safety-critical scenarios. As most robots and autonomous vehicles make heavy use of cameras to perceive their surroundings, camera modeling becomes a prerequisite for the successful simulation of these autonomous agents. This contribution outlines the context in which camera models are used; provides a component-by-component analysis of the image acquisition pipeline along with algorithms used for its modeling; and closes with a discussion of data-driven approaches that embrace a different perspective on camera modeling.

We describe a simulation environment that enables the design and testing of control policies for off-road mobility of autonomous agents. The environment is demonstrated in conjunction with the training and assessment of a reinforcement learning policy that uses sensor fusion and interagent communication to enable the movement of mixed convoys of human-driven and autonomous vehicles. Policies learned on rigid terrain are shown to transfer to hard (silt-like) and soft (snow-like) deformable terrains. The environment described performs the following: multivehicle multibody dynamics cosimulation in a time/space-coherent infrastructure that relies on the Message Passing Interface standard for low-latency parallel computing; sensor simulation (e.g., camera, GPU, IMU); simulation of a virtual world that can be altered by the agents present in the simulation; training that uses reinforcement learning to “teach” the autonomous vehicles to drive in an obstacle-riddled course. The software stack described is open source. Relevant movies: Project Chrono. Off-road AV simulations, 2020².