Wenxuan Wang

Wenxuan Wang

Auto-generated profile, based on publicly available information

Wenxuan Wang

School of Vehicle and Mobility, Tsinghua University, Beijing, China

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 January 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
JDJingliang Duan
Jingliang Duan
WWWenxuan Wang
Wenxuan Wang
LXLiming Xiao
Liming Xiao
JGJiaxin Gao
Jiaxin Gao
SLShengbo Li
Shengbo Li
KLKeqiang Li
Keqiang Li
Reinforcement learning (RL) has shown remarkable success in solving complex decision-making and control tasks. However, many model-free RL algorithms experience performance degradation due to inaccurate value estimation, particularly the overestimation of Q-values, which can lead to suboptimal policies. To address this issue, we previously proposed the Distributional Soft Actor-Critic (DSAC or DSACv1), an off-policy RL algorithm that enhances value estimation accuracy by learning a continuous Gaussian value distribution. Despite its effectiveness, DSACv1 faces challenges such as training instability and sensitivity to reward scaling, caused by high variance in critic gradients due to return randomness. In this paper, we introduce three key refinements to DSACv1 to overcome these limitations and further improve Q-value estimation accuracy: expected value substitution, twin value distribution learning, and variance-based critic gradient adjustment. The enhanced algorithm, termed DSAC with Three refinements (DSAC-T or DSACv2), is systematically evaluated across a diverse set of benchmark tasks. Without the need for task-specific hyperparameter tuning, DSAC-T consistently matches or outperforms leading model-free RL algorithms, including SAC, TD3, DDPG, TRPO, and PPO, in all tested environments. Additionally, DSAC-T ensures a stable learning process and maintains robust performance across varying reward scales. Its effectiveness is further demonstrated through real-world application in controlling a wheeled robot, highlighting its potential for deployment in practical robotic tasks.

17 April 2023 in Communications in Transportation Research
Verified
Journal Article
Open access
WWWenxuan Wang
Wenxuan Wang
YZYuhang Zhang
Yuhang Zhang
JGJiaxin Gao
Jiaxin Gao
YJYuxuan Jiang
Yuxuan Jiang
YYYujie Yang
Yujie Yang
SLShengbo Li
Shengbo Li

24 December 2021 in IEEE Transactions on Intelligent Transportation Systems
Verified
Journal Article
Closed access
JDJingliang Duan
Jingliang Duan
DYDongjie Yu
Dongjie Yu
SLShengbo Li
Shengbo Li
WWWenxuan Wang
Wenxuan Wang
YRYangang Ren
Yangang Ren
BCBo Cheng
Bo Cheng
In this paper, we propose a new state representation method, called encoding sum and concatenation (ESC), to describe the environment observation for decision-making in autonomous driving. Unlike existing state representation methods, ESC is applicable to the situation where the number of surrounding vehicles is variable and eliminates the need for manually pre-designed sorting rules, leading to higher representation ability and generality. The proposed ESC method introduces a feature neural network (NN) to encode the real-valued feature of each surrounding vehicle into an encoding vector, and then adds these vectors up to obtain the representation vector of the set of surrounding vehicles. Then, a fixed-dimensional and permutation-invariance state representation can be obtained by concatenating the set representation with other variables, such as indicators of the ego vehicle and road. By introducing the sum-of-power mapping, this paper has further proved that the injectivity of the ESC state representation can be guaranteed if the output dimension of the feature NN is greater than the number of variables of all surrounding vehicles. This means that the ESC representation can be used to describe the environment and taken as the inputs of learning-based policy functions. Experiments demonstrate that compared with the fixed-permutation representation method, the policy learning accuracy based on ESC representation is improved by 62.2%.

21 August 2024 in IEEE Transactions on Intelligent Vehicles
Verified
Journal Article
Closed access
WWWenxuan Wang
Wenxuan Wang
JDJingliang Duan
Jingliang Duan
XSXujie Song
Xujie Song
LXLiming Xiao
Liming Xiao
LCLiangfa Chen
Liangfa Chen
SLShengbo Li
Shengbo Li
Reinforcement learning (RL) has demonstrated considerable potential in addressing intricate control and decision problems such as vehicle tracking control and obstacle avoidance. Nonetheless, the control policies acquired through RL often lack smoothness even in the presence of minor noises or disturbances, which may induce oscillations, overheating, and even damage to the system in real-world applications. Existing methods handle this issue from temporal or spatial domains and are often tightly coupled with specific tasks. This paper studies the smoothness of control policy from the frequency domain perspective. We propose a class of neural networks with low-pass filtering ability, named Smonet, to alleviate the non-smooth issue by learning a low-frequency representation within hidden layers. Smonet features with serial filtering layers responsible for low-pass filtering of the input signal. Each filtering layer contains multiple inertia cells, one adaptive cell, and one activation layer. To facilitate the filtering ability of Smonet, we further proposed a Smonet-based RL training method by integrating an extra regularization term relating to filtering factors to standard RL loss. Finally, we assess the efficacy of Smonet through diverse simulated robot control tasks and a real-world mobile robot obstacle avoidance experiment, comparing its performance with two commonly utilized networks, multi-layer perceptron and gated recurrent unit. Results indicate that Smonet consistently enhances policy smoothness under various observation noises without compromising control performance. Notably, it achieves up to a 72.7% reduction in the action fluctuation ratio compared to traditional network structures.

19 March 2026 in Expert Systems with Applications
Verified
Journal Article
Closed access
LCLiangfa Chen
Liangfa Chen
XSXujie Song
Xujie Song
JZJunjie Zhao
Junjie Zhao
WWWenxuan Wang
Wenxuan Wang
YWYinuo Wang
Yinuo Wang
JDJingliang Duan
Jingliang Duan

4 October 2024
Verified
Book Chapter
Open access
PDF file
JGJiaxin Gao
Jiaxin Gao
YLYao Lyu
Yao Lyu
WWWenxuan Wang
Wenxuan Wang
YYYuming Yin
Yuming Yin
Fei MaFei Ma
Fei Ma
SLShengbo Li
Shengbo Li
Distributed stochastic gradient descent techniques have gained significant attention in recent years as a prevalent approach for reinforcement learning. Current distributed learning predominantly employs synchronous or asynchronous training strategies. While the asynchronous scheme avoids idle computing resources present in synchronous methods, it grapples with the stale gradient issue. This paper introduces a novel gradient correction algorithm aimed at alleviating the stale gradient problem. By leveraging second-order information within the worker node and incorporating current parameters from both the worker and server nodes, the gradient correction algorithm yields a refined gradient closer to the desired value. Initially, we outline the challenges associated with asynchronous update schemes and derive a gradient correction algorithm employing local second-order approximations. Subsequently, we propose an asynchronous training scheme incorporating gradient correction within the generalized policy iteration framework. Lastly, in the context of trajectory tracking tasks, we compare the impact of employing gradient correction versus its absence in an asynchronous update scheme. Simulation results underscore the superiority of our proposed training scheme, demonstrating notably faster convergence and higher policy performance compared to the existing asynchronous update methods.