Changqiang Huang

Changqiang Huang

Auto-generated profile, based on publicly available information

Changqiang Huang

Aeronautics Engineering College, Air Force Engineering University, Xi’an 710038, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
11 October 2022 in Applied Sciences
Verified
Journal Article
Open access
PDF file
HZHongpeng Zhang
Hongpeng Zhang
YWYujie Wei
Yujie Wei
HZHuan Zhou
Huan Zhou
CHChangqiang Huang
Changqiang Huang
Maneuver decision-making is the core of autonomous air combat, and reinforcement learning is a potential and ideal approach for addressing decision-making problems. However, when reinforcement learning is used for maneuver decision-making for autonomous air combat, it often suffers from awful training efficiency and poor performance of maneuver decision-making. In this paper, an air combat maneuver decision-making method based on final reward estimation and proximal policy optimization is proposed to solve the above problems. First, an air combat environment based on aircraft and missile models is constructed, and an intermediate reward and final reward are designed. Second, the final reward estimation is proposed to replace the original advantage estimation function of the surrogate objective of proximal policy optimization to improve the training performance of reinforcement learning. Third, sampling according to the final reward estimation is proposed to improve the training efficiency. Finally, the proposed method is used in a self-play framework to train agents for maneuver decision-making. Simulations show that final reward estimation and sampling according to final reward estimation are effective and efficient.

9 October 2018 in Neural Computing and Applications
Verified
Journal Article
Closed access
PDF file
ZZZhuoran Zhang
Zhuoran Zhang
HHHanqiao Huang
Hanqiao Huang
CHChangqiang Huang
Changqiang Huang
BHBo Han
Bo Han

13 January 2020 in IEEE Access
Verified
Journal Article
Open access
HZHongpeng Zhang
Hongpeng Zhang
CHChangqiang Huang
Changqiang Huang
Maneuver decision-making directly determines the success or failure of air combat. To improve the dogfight ability of unmanned combat aerial vehicles and avoid the deficiencies of traditional methods, such as poor flexibility and a weak decision-making ability, a maneuver method using deep learning is proposed. A total of 72 different maneuvers are constructed, and 544320 states are designed. Flight simulations are conducted under these different states to obtain corresponding future azimuth angles. A deep neural network is trained with these offline data, and thus, the network possesses state prediction capability. A situation assessment function and a decision objective function based on azimuth angles are constructed. During air combat, the optimal maneuver is selected from the maneuver library according to the predicted state and the decision objective function. The results of air combat simulations indicate that the unmanned combat aerial vehicle (UCAV) can win the air combat game by the proposed method in a balanced situation and can meet missile launching conditions in an adverse situation. The operational time of this method has been reduced by 0.01 s compared with the comparison method.

25 October 2022 in Frontiers in Neurorobotics
Verified
Journal Article
Open access
HZHongpeng Zhang
Hongpeng Zhang
HZHuan Zhou
Huan Zhou
YWYujie Wei
Yujie Wei
CHChangqiang Huang
Changqiang Huang
Autonomous maneuver decision-making methods for air combat often rely on human knowledge, such as advantage functions, objective functions, or dense rewards in reinforcement learning, which limits the decision-making ability of unmanned combat aerial vehicle to the scope of human experience and result in slow progress in maneuver decision-making. Therefore, a maneuver decision-making method based on deep reinforcement learning and Monte Carlo tree search is proposed to investigate whether it is feasible for maneuver decision-making without human knowledge or advantage function. To this end, Monte Carlo tree search in continuous action space is proposed and neural networks-guided Monte Carlo tree search with self-play is utilized to improve the ability of air combat agents. It starts from random behaviors and generates samples consisting of states, actions, and results of air combat through self-play without using human knowledge. These samples are used to train the neural network, and the neural network with a greater winning rate is selected by simulations. Then, repeat the above process to gradually improve the maneuver decision-making ability. Simulations are conducted to verify the effectiveness of the proposed method, and the kinematic model of the missile is used in simulations instead of the missile engagement zone to test whether the maneuver decision-making method is effective or not. The simulation results of the fixed initial state and random initial state show that the proposed method is efficient and can meet the real-time requirement.

19 August 2023 in Applied Sciences
Verified
Journal Article
Open access
PDF file
YWYujie Wei
Yujie Wei
HZHongpeng Zhang
Hongpeng Zhang
YWYuan Wang
Yuan Wang
CHChangqiang Huang
Changqiang Huang
Maneuver decision-making is essential for autonomous air combat. However, previous methods usually make decisions to aim at the target instead of hitting the target and use discrete action spaces instead of continuous action spaces. While these simplifications make maneuver decision-making easier, they also make maneuver decision-making more unrealistic. Meanwhile, previous studies usually rely on handcrafted reward functions, which are troublesome to design. Therefore, to solve these problems, we propose an automatic curriculum reinforcement learning method that enables agents to maneuver effectively in air combat from scratch. On the basis of curriculum reinforcement learning, maneuver decision-making is divided into a series of sub-tasks from easy to difficult. Thus, agents can gradually learn how to complete a series of sub-tasks, from easy to difficult without handcrafted reward functions. The ablation studies show that automatic curriculum learning is essential for reinforcement learning; namely, agents cannot make effective decisions without curriculum learning. Simulations show that, after training, agents are able to make effective decisions given different states, including tracking, attacking, and escaping, which are both rational and interpretable.

19 July 2023 in IEEE Access
Verified
Journal Article
Open access
YWYujie Wei
Yujie Wei
HZHongpeng Zhang
Hongpeng Zhang
YWYuan Wang
Yuan Wang
CHChangqiang Huang
Changqiang Huang
Reinforcement learning is an effective approach for solving decision-making problems. However, when using reinforcement learning to solve maneuver decision-making with sparse rewards, it costs too much time for training, and the final performance may not be satisfactory. In order to overcome the shortcomings, the method for maneuver decision-making based on curriculum learning and reinforcement learning is proposed. First, three curricula are designed to address the maneuver decision-making problem: angle curriculum, distance curriculum and hybrid curriculum. They are proposed according to the intuition that closer destinations are easier to arrive at. Then, they are used to train agents and compared with the original method without any curriculum. The training results show that angle curriculum can increase the speed and stability of training, and improve the performance of maneuver decision-making; distance curriculum can increase the speed and stability of agent training; hybrid curriculum is not better than the other curricula, because it makes the agent get stuck at the local optimum. The simulation results show that after training, the agent can handle the situations where targets come from different directions, and the maneuver decision-makings are rational, effective, and interpretable, whereas the method without curriculum is invalid.