Humans drive in a holistic fashion which entails, in particular, understanding dynamic road events and their evolution. Injecting these capabilities in autonomous vehicles can thus take situational awareness and decision making closer to human-level performance. To this purpose, we introduce the ROad event Awareness Dataset (ROAD) for Autonomous Driving, to our knowledge the first of its kind. ROAD is designed to test an autonomous vehicle’s ability to detect road events, defined as triplets composed by an active agent, the action(s) it performs and the corresponding scene locations. ROAD comprises videos originally from the Oxford RobotCar Dataset, annotated with bounding boxes showing the location in the image plane of each road event. We benchmark various detection tasks, proposing as a baseline a new incremental algorithm for online road event awareness termed 3D-RetinaNet. We also report the performance on the ROAD tasks of Slowfast and YOLOv5 detectors, as well as that of the winners of the ICCV2021 ROAD challenge, which highlight the challenges faced by situation awareness in autonomous driving. ROAD is designed to allow scholars to investigate exciting tasks such as complex (road) activity detection, future event anticipation and continual learning. The dataset is available at https://github.com/gurkirt/road-dataset ; the baseline can be found at https://github.com/gurkirt/3D-RetinaNet .

Using robotic models to test theories on the behavior of humans or animals can help understand many aspects of intelligence and control in natural systems. In this paper, we study how model-based optimal control principles can explain stereotyped human oculomotor behaviors, through simulations in a realistic model of the human eye with a cable-driven actuation system that mimics the six degrees of freedom of the extraocular muscles. Previous works have only addressed systems with 1-3 degrees-of-freedom. The current paper is a first study on a six-muscle system design, which introduces novel challenges to address. We propose nonlinear optimal control techniques to optimize the accuracy, energy, and duration of eye movement trajectories. We use a recurrent neural network that learns to emulate the nonlinear system dynamics from recorded sample trajectories. We focus on the generation of rapid saccadic eye movements with fully unconstrained kinematics and corresponding control signals for the six cables that simultaneously satisfied the proposed optimization criteria. We show that realistic three-dimensional rotational kinematics and dynamics, as seen in human saccades, emerged from our model. Interestingly, just as in the primate oculomotor system, the six cables organized themselves into appropriate antagonistic muscle pairs without slack.

In human-robot interactions, eye movements play an important role in non-verbal communication. However, controlling the motions of a robotic eye that display similar performance as the human oculomotor system is still a major challenge. In this paper, we study how to control a realistic model of the human eye with a cable-driven actuation system that mimics the six degrees of freedom of the extra-ocular muscles. The biomimetic design introduces novel challenges to address, most notably the need to control the pretension on each individual muscle to prevent the loss of tension during motion, that would lead to cable slack and lack of control. We built a robotic prototype and developed a nonlinear simulator and two controllers. In the first approach, we linearized the nonlinear model, using a local derivative technique, and designed linear-quadratic optimal controllers to optimize a cost function that accounts for accuracy, energy expenditure, and movement duration. The second method uses a recurrent neural network that learns the nonlinear system dynamics from sample trajectories of the system, and a non-linear trajectory optimization solver that minimizes a similar cost function. We focused on the generation of rapid saccadic eye movements with fully unconstrained kinematics, and the generation of control signals for the six cables that simultaneously satisfied several dynamic optimization criteria. The model faithfully mimics the three-dimensional rotational kinematics and dynamics observed for human saccades. Our experimental results indicate that while both methods yielded similar results, the nonlinear method is more flexible for future improvements to the model, for which the calculations of the linearized model's position-dependent pretensions and local derivatives become particularly tedious.