In 3 vs. 3 online basketball games, finite state machine (FSM)-based Game artificial intelligence (AI) has traditionally been employed. However, limitations such as repetitive behavior patterns and challenges in maintaining systems during redesigns have led to increased research into reinforcement learning-based AI. This shift aims to address the shortcomings of FSM-based approaches. Nevertheless, applying multi-agent reinforcement learning-based AI in commercial online basketball games presents significant challenges, particularly in ensuring real-time processing, which requires efficient methods for managing observational data. In addition, the stochastic nature of action selection in reinforcement learning complicates the accurate learning of behaviors through explicit decision data. Moreover, reinforcement learning, which self-optimizes through exploration and develops its own rules, struggles to mimic human-like behavior patterns that follow predefined strategies in Sports Game. This study introduces a human strategy-based reinforcement learning method designed to address these challenges and replicate human gameplay that adheres to human-defined strategies. The learning of human strategies is enhanced using Ray for the real-time processing of observational data and a multi-phase reward system that distinctly defines rewards based on specific objectives. Furthermore, the proposed method enables real-time, strategy-based action guidance through a Human Strategy AI trained on human-defined strategies. Experimental results demonstrate that in a stochastic basketball game environment, this approach enabled the determination of precise actions and achieved human-like gameplay through the Human Strategy AI.
In this paper, we propose an advanced double layered multi-agent system to reduce learning time, expressing a state space using a 2D grid. This system is based on asynchronous advantage actor-critic systems (A3C) and reduces the state space that agents need to consider by hierarchically expressing a 2D grid space and determining actions. Specifically, the state space is expressed in the upper and lower layers. Based on the learning results using A3C in the lower layer, the upper layer makes decisions without additional learning, and accordingly, the total learning time can be reduced. Our method was verified experimentally using a virtual autonomous surface vehicle simulator. It reduced the learning time required to reach a 90% goal achievement rate by 7.1% compared to the conventional double layered A3C. In addition, the goal achievement by the proposed method was 18.86% higher than that of the traditional double layered A3C over 20,000 learning episodes.
Considering autonomous navigation in busy marine traffic environments (including harbors and coasts), major study issues to be solved for autonomous ships are avoidance of static and dynamic obstacles, surface vehicle control in consideration of the environment, and compliance with human-defined navigation rules. The reinforcement learning (RL) algorithm, which demonstrates high potential in autonomous cars, has been presented as an alternative to mathematical algorithms and has advanced in studies on autonomous ships. However, the RL algorithm, through interactions with the environment, receives relatively fewer data from the marine environment. Moreover, the open marine environment causes difficulties for autonomous ships in learning human-defined navigation rules because of excessive degrees of freedom. This study proposes a sustainable, intelligent learning framework for autonomous ships (ILFAS), which helps solve these difficulties and learns navigation rules specified by human beings through neighboring ships. The application of case-based RL enables the participation of humans in the RL learning process through neighboring ships and the learning of human-defined rules. Cases built as curriculums can achieve high learning effects with fewer data along with the RL of layered autonomous ships. The experiment aims at autonomous navigation from a harbor, where marine traffic occurs on a neighboring coast. The learning results using ILFAS and those in an environment where random marine traffic occurs are compared. Based on the experiment, the learning time was reduced by a tenth. Moreover, the success rate of arrival at a destination was higher with fewer controls than the random method in the new marine traffic scenario. ILFAS can continuously respond to advances in ship manufacturing technology and changes in the marine environment.
Monocular three-dimensional (3D) scene understanding tasks, e.g., object size angle and 3D position, estimation are challenging to perform. More successful current methods usually require data from 3D sensors, which limits their performance if only monocular images are used because of the lack of distance information. In recent years, 3D object detection methods based on pseudo-LiDAR have effectively improved the accuracy of 3D prediction. However, most pseudo-LiDAR methods directly use LiDAR-based 3D detection networks, which also limits their performance. In this paper, a new monocular 3d object detection framework is proposed. By redesigning the representation of pseudo-LiDAR, a 2d detection mask channel is introduced as a guide layer to guide 3D object detection. A new 3D object detection network is designed for the newly designed multi-channel data. The pillar feature encoding module is optimized by using 2d detection mask and height histogram, so that it can more effectively convert point clouds into pillar features by using 2d detection mask and height characteristics of different objects. A new feature extraction network was designed using the transformer module to extract pillar features. The transformer head was optimized using three independent heads to categorize the position, properties and specific parameters of the object. An evaluation on the challenging KITTI dataset demonstrates that the proposed method significantly improves the performance of state-of-the-art monocular methods.
Monocular depth prediction research is essential for expanding meaning from 2D to 3D. Recent studies have focused on the application of a newly proposed encoder; however, the development within the self-supervised learning framework remains unexplored, an aspect critical for advancing foundational models of 3D semantic interpretation. Addressing the dynamic nature of encoder-based research, especially in performance evaluations for feature extraction and pre-trained models, this research proposes the switchable encoder learning framework (SELF). SELF enhances versatility by enabling the seamless integration of diverse encoders in a self-supervised learning context for depth prediction. This integration is realized through the direct transfer of feature information from the encoder and by standardizing the input structure of the decoder to accommodate various encoder architectures. Furthermore, the framework is extended and incorporated into an adaptable decoder for depth prediction and camera pose learning, employing standard loss functions. Comparative experiments with previous frameworks using the same encoder reveal that SELF achieves a 7% reduction in parameters while enhancing performance. Remarkably, substituting newly proposed algorithms in place of an encoder improves the outcomes as well as significantly decreases the number of parameters by 23%. The experimental findings highlight the ability of SELF to broaden depth factors, such as depth consistency. This framework facilitates the objective selection of algorithms as a backbone for extended research in monocular depth prediction.
Establishing a framework capable of performing 3D object tracking is crucial for various applications, such as autonomous driving and robot navigation. Monocular cameras offer economic and flexible advantages over LiDAR sensors; however, monocular camera-based 3D object detection is fraught with uncertainty owing to difficulty in depth estimation, visual occlusion, and ambiguous object appearance. This uncertainty poses additional challenges to object tracking as well, making performing stable multiple-object tracking on monocular 3D detection results more complex and difficult. To address these challenges in monocular 3D multiple-object detection and tracking, we propose the innovative framework pseudo-LiDAR-MOT, which accurately infers complete 3D bounding box information from 2D image sequences captured by a monocular camera and efficiently correlates and tracks moving objects in the temporal dimension. This framework increases the object detection accuracy and tracking stability of objects in dynamic environments via the following three methods. Monocular depth prediction and 2D object detection networks are leveraged to estimate pixelwise depth and object boundaries. Subsequently, a pseudo-LiDAR approach is employed to generate a 3D point cloud for each object, where 3D point clouds are transformed into 2D representations. Furthermore, a novel 3D detection network based on image-based detection is introduced, simplifying the complexity of conventional 3D point-cloud-based detection while increasing the accuracy of monocular image-based 3D object detection. A specialized association network is designed to enhance object trajectory tracking across frames, thereby improving 3D tracking performance. The effectiveness of our approach was validated through extensive experiments conducted on the KITTI dataset. Our proposed framework exhibits excellent performance in 3D object detection and tracking, outperforming conventional methods. This study introduces innovative ideas for 3D object detection and tracking and establishes a basis for advancing autonomous driving systems.