Human demonstrations can provide trustful samples to train reinforcement learning algorithms for robots to learn complex behaviors in real-world environments. However, obtaining sufficient demonstrations may be impractical because many behaviors are difficult for humans to demonstrate. A more practical approach is to replace human demonstrations by human queries, i.e., preference-based reinforcement learning. One key limitation of the existing algorithms is the need for a significant amount of human queries because a large number of labeled data is needed to train neural networks for the approximation of a continuous, high-dimensional reward function. To reduce and minimize the need for human queries, we propose a new GAN-assisted human preference-based reinforcement learning approach that uses a generative adversarial network (GAN) to learn human preferences and then replace the role of human in assigning preferences. The adversarial neural network is simple and only has a binary output, hence requiring much less human queries to train. Moreover, a maximum entropy based reinforcement learning algorithm is designed to shape the loss towards the desired regions or away from the undesired regions. To show the effectiveness of the proposed approach, we present some studies on complex robotic tasks without access to the environment reward in a typical MuJoCo robot locomotion environment. The obtained results show our method can achieve a reduction of about 99.8% human time without performance sacrifice.
1 December 2005 in Journal of Central South University of Technology
Entropy regularization is a commonly used technique in reinforcement learning to improve exploration and cultivate a better pre-trained policy for later adaptation. Recent studies further show that the use of entropy regularization can smooth the optimization landscape and simplify the policy optimization process, indicating the value of integrating entropy into reinforcement learning. However, existing studies only consider the policy's entropy at the current state as an extra regularization term in the policy gradient or in the objective function without formally integrating the entropy in the reward function. In this paper, we propose a shaped reward that includes the agent's policy entropy into the reward function. In particular, the agent's expected entropy over a distribution of the next state is added to the immediate reward associated with the current state. The addition of the agent's expected policy entropy at the next state distribution is shown to yield new soft Q function and state function that are concise and modular. Moreover, the new reinforcement learning framework can be easily applied to the existing standard reinforcement learning algorithms, such as DQN and PPO, while inheriting the benefits of employing entropy regularization. We further present a soft stochastic policy gradient theorem based on the shaped reward and propose a new practical reinforcement learning algorithm. Finally, a few experimental studies are conducted in MuJoCo environment to demonstrate that our method can outperform an existing state-of-the-art off-policy maximum entropy reinforcement learning approach Soft Actor-Critic by 5-150% in terms of average return.
18 June 2019 in IEEE Transactions on Emerging Topics in Computational Intelligence
The resistance of computational models against label noise offers promising potential for the correction of erroneous labels. One intuitive way is to re-label the data samples based on the model’s prediction when the original label error rate is relatively high. However, directly flipping labels to the model’s prediction may not improve the quality of datasets due to the concentration of noisy labels at small regions, i.e., increasing noise condensity, in the process of label flipping. Given the same label accuracy, datasets with condensed noise lead to (much) worse learning models than those without condensed noise. Hence, the quality of dataset may not benefit from this correction process. Moreover, iteratively flipping the label typically leads to the decreasing of label accuracy rather than a stabilized error rate around which the error rate slowly oscillates at each subsequent iteration. In this paper, we propose a novel method that simultaneously reduces the label error rate and improves the quality of datasets (via reduction of noise condensity). In contrast to the existing methods that either involve humans in the label correction process or construct multiple models to obtain a consensus opinion, our proposed method is simple and can automatically improve the quality of datasets. Specifically, we propose the use of a small clean dataset to evaluate overfitting caused by the concentration of label noise. Once the noise condensity issue is detected, the label flipping process will be modified by adding a statistical probability into the flipping procedure. The proposed method is verified on noisy MNIST and CIFAR-10 datasets. Label correction results are presented and the prediction accuracies of the neural network model trained on the corrected datasets are compared with results from the methods that target for learning from noisy labels.
Based on the sound intensity measurement and analyzing, the noise radiated from the washing machine with horizontal axis was discussed to identify the noise source and find its propagation path. According to the principle of the sound intensity measurement, the measurement grids were arranged at the outside surfaces of the washer, and the distance was set to 20cm. The result showed that the noise source was near the right surface of washer’s motor installing. The maximum sound intensity values are mainly in frequency bands of 2 KHz and 4 KHz. Back surface of washer with sound power level 51 dB radiated the maximum sound intensity value 55.8 dB for 2 KHz and 56.0 dB for 4 KHz. There was also appeared the maximum sound intensity of low frequency 100 Hz in Right surface, which was caused by the vibration of motor transferring to the right surface shell. The results can provide technology support for the noise control of washing machine.
10 August 2018 in IEEE Transactions on Industrial Informatics
This paper investigates the problem of extracting actionable patterns/models from unlabeled and erroneous datasets in an unsupervised way. We propose a novel iterative multi-layer micro-macro method (IM3) that defines data reliability, learns micro-macro models, and refines learned models iteratively. The IM3 method includes a general data reliability definition, a micro-macro model complexity determination, and an iterative data reliability and model complexity update. In particular, we propose a consistency index based approach to address underfitting and overfitting in an unsupervised way. The refinement of the learned models is enabled via dropping the most unreliable data until the data reliability is above a given threshold. The sensitivity of the proposed IM3 method with respect to the reliability threshold selection is further quantified via false alarm and miss detection. Evaluation of the proposed method and quantitative analysis of its sensitivity are provided on a polynomial regression problem via Monte Carlo simulations.