We demonstrate a working functional prototype of a cooperative perception system that maintains a real-time digital twin of the traffic environment, providing a more accurate and more reliable model than any of the participant subsystems—in this case, smart vehicles and infrastructure stations—would manage individually. The importance of such technology is that it can facilitate a spectrum of new derivative services, including cloud-assisted and cloud-controlled ADAS functions, dynamic map generation with analytics for traffic control and road infrastructure monitoring, a digital framework for operating vehicle testing grounds, logistics facilities, etc. In this paper, we constrain our discussion on the viability of the core concept and implement a system that provides a single service: the live visualization of our digital twin in a 3D simulation, which instantly and reliably matches the state of the real-world environment and showcases the advantages of real-time fusion of sensory data from various traffic participants. We envision this prototype system as part of a larger network of local information processing and integration nodes, i.e., the logically centralized digital twin is maintained in a physically distributed edge cloud.

The paper focuses on the problem of raw data fusion in neural networks based 3D object detection architectures. Here we consider the case of autonomous driving with data from camera and LiDAR sensors. Understanding the vehicle surroundings is a crucial task in autonomous driving since any subsequent action taken is strongly dependent on it. In this paper we present an alternative method of fusing camera image information with LiDAR poinclouds at a close to raw level of abstraction. Our results suggest that our approach improves the average precision of 3D bounding box detection of cyclists (and possibly other objects) in sparse point clouds compared to the baseline architecture without low-level fusion. The proposed approach has been evaluated on the KITTI dataset containing driving scenes with corresponding camera and LiDAR data. The long-term goal of our research is to develop a neural network architecture for environment perception that fuses multi-sensor data at the earliest stages possible, thus leveraging the full benefits of possible inter-sensor synergies.

Adversarial attacks threaten the reliability of machine learning models in critical applications like autonomous vehicles and defense systems. As object detectors become more robust with models like YOLOv8, developing effective adversarial methodologies is increasingly challenging. We present Truck Adversarial Camouflage Optimization (TACO), a novel framework that generates adversarial camouflage patterns on 3D vehicle models to deceive state-of-the-art object detectors. Adopting Unreal Engine 5, TACO integrates differentiable rendering with a Photorealistic Rendering Network to optimize adversarial textures targeted at YOLOv8. To ensure the generated textures are both effective in deceiving detectors and visually plausible, we introduce the Convolutional Smooth Loss function, a generalized smooth loss function. Experimental evaluations demonstrate that TACO significantly degrades YOLOv8’s detection performance, achieving an AP@0.5 of 0.0099 on unseen test data. Furthermore, these adversarial patterns exhibit strong transferability to other object detection models such as Faster R-CNN and earlier YOLO versions.

Autonomous driving functions increasingly incorporate machine learning methods for perception, environment modeling and decision making. The ever-present danger of unintended behavior of neural networks has given cause for concern in safety-critical applications and has spurred research towards the establishment of certified properties. In our work we present a general approach for exploring guaranteed properties of neural networks through a specific use case. We first methodically specified several desired properties for an ideal traffic sign classifier. Then we applied a mixed integer linear programming verification method to certify (or conversely, falsify) that our pre-trained rectified linear unit (ReLU) activation network will satisfy given properties under all conditions. We established through relaxations of our initial specification that our model does in fact happen to guarantee certain (but not all) properties at a different threshold. We thus showed that it is possible to establish the degree to which non-trivial and humanly meaningful properties are inherently guaranteed by an arbitrary pixel space ReLU network. We believe this approach can be a helpful addition in comprehensively evaluating the reliability of a network via quantifying its degree of compliance to an ideal specification.

Activity Chain Optimization (ACO) is the task of finding a minimum-cost tour that visits exactly one location for each required activity while respecting time window constraints. We develop an exact algorithm that efficiently solves the ACO problem in all practical cases that involve hundreds of locations offering up to 10–15 activities and returns the optimal route with minimal time spent traveling and waiting. We also introduce a greedy heuristic that simulates human decision-making for comparison. Our experimental results highlight the practical significance of our work as we can reduce travel and wait times on 45 realistic Budapest inner-city routing problems by 16.65% on average compared to our baseline. Our algorithms’ computational and memory requirements for solving practical ACO instances are shown to be low enough to be employed on embedded devices, e.g. smartphones and navigation systems.

Border surveillance and the monitoring of critical infrastructure are essential components of regional and industrial security. In this paper, our purpose is to study the intricate nature of surveillance methods used by hybrid monitoring systems utilizing Pan–Tilt–Zoom (PTZ) cameras, modeled as directional sensors, and UAVs. We aim to accomplish three occasionally conflicting goals. Firstly, at any given moment we want to detect as many intruders as possible with special attention to newly arriving trespassers. Secondly, we consider it equally important to observe the temporal movement and behavior of each intruder group as accurately as possible. Furthermore, in addition to these objectives, we also seek to minimize the cost of sensor usage associated with surveillance. During the research, we developed and analyzed several interrelated, increasingly complex algorithms. By leveraging RL methods we also gave the system the chance to find the optimal solution on its own. As a result we have gained valuable insights into how various components of these algorithms are interconnected and coordinate. Building upon these observations, we managed to develop an efficient algorithm that takes into account all three criteria mentioned above.