Regularization is a popular technique in machine learning for model estimation and for avoiding overfitting. Prior studies have found that modern ordered regularization can be more effective in handling highly correlated, high-dimensional data than traditional regularization. The reason stems from the fact that the ordered regularization can reject irrelevant variables and yield an accurate estimation of the parameters. How to scale up the ordered regularization problems when facing large-scale training data remains an unanswered question. This paper explores the problem of parameter estimation with the ordered -regularization via Alternating Direction Method of Multipliers (ADMM), called ADMM-O . The advantages of ADMM-O include (i) scaling up the ordered to a large-scale dataset, (ii) predicting parameters correctly by excluding irrelevant variables automatically, and (iii) having a fast convergence rate. Experimental results on both synthetic data and real data indicate that ADMM-O can perform better than or comparable to several state-of-the-art baselines.
Off-policy learning exhibits greater instability when compared to on-policy learning in reinforcement learning (RL). The difference in probability distribution between the target policy (\(\pi\)) and the behavior policy (b) is a major cause of instability. High variance also originates from distributional mismatch. The variation between the target policy’s distribution and the behavior policy’s distribution can be reduced using importance sampling (IS). However, importance sampling has high variance, which is exacerbated in sequential scenarios. We propose a smooth form of importance sampling, specifically relative importance sampling (RIS), which mitigates variance and stabilizes learning. To control variance, we alter the value of the smoothness parameter \(\beta \in [0, 1]\) in RIS. We develop the first model-free relative importance sampling off-policy actor-critic (RIS-off-PAC) algorithms in RL using this strategy. Our method uses a network to generate the target policy (actor) and evaluate the current policy (\(\pi\)) using a value function (critic) based on behavior policy samples. Our algorithms are trained using behavior policy action values in the reward function, not target policy ones. Both the actor and critic are trained using deep neural networks. Our methods performed better than or equal to several state-of-the-art RL benchmarks on OpenAI Gym challenges and synthetic datasets.
In numerous episodic reinforcement learning (RL) environments, SARSA-based methodologies are employed to enhance policies aimed at maximizing returns over long horizons. Traditional SARSA algorithms face challenges in achieving an optimal balance between bias and variation, primarily due to their dependence on a single, constant discount factor (). This study enhances the temporal difference decomposition method, TD(), by applying it to the SARSA algorithm, wherein the action-value function is segmented into several components based on the differences between action-value functions linked to specific discount factors. Each component, referred to as a delta estimator (D), is linked to a specific discount factor and learned independently. This modified technique is referred to as SARSA(). SARSA is a widely used on-policy RL method that enhances action-value functions via temporal difference updates. This decomposition, namely SARSA(), facilitates learning across a range of time scales. This analysis makes learning more effective and guarantees consistency, especially in situations where long-horizon improvement is needed. The results of this research show that the proposed technique works to lower bias in SARSA’s updates and speed up convergence in both deterministic and stochastic settings, even in dense-reward Atari environments. Experimental results from a variety of benchmark settings show that the proposed SARSA() outperforms existing TD learning techniques in both tabular and deep RL environments.
1 January 2014 in Lecture Notes in Computer Science
Off-policy learning is more unstable compared to on-policy learning in reinforcement learning (RL). One reason for the instability of off-policy learning is a discrepancy between the target ($\pi$) and behavior (b) policy distributions. The discrepancy between $\pi$ and b distributions can be alleviated by employing a smooth variant of the importance sampling (IS), such as the relative importance sampling (RIS). RIS has parameter $\beta\in[0, 1]$ which controls smoothness. To cope with instability, we present the first relative importance sampling-off-policy actor-critic (RIS-Off-PAC) model-free algorithms in RL. In our method, the network yields a target policy (the actor), a value function (the critic) assessing the current policy ($\pi$) using samples drawn from behavior policy. We use action value generated from the behavior policy in reward function to train our algorithm rather than from the target policy. We also use deep neural networks to train both actor and critic. We evaluated our algorithm on a number of Open AI Gym benchmark problems and demonstrate better or comparable performance to several state-of-the-art RL baselines.
Automatic selection of true explanatory variables and controlling fraction of false discovery rate (FDR) in the linear model has received considerable attention in machine learning. The ordered regularization is an important component of the linear model and plays a key role in solving such kind of problems. Although there have been some models proposed for determining relevant features in either low-dimensional or high-dimensional space. However, there exists no single sorted model that can work in both low-high dimensions cases. This paper introduces a model called mSLOPE (model-free SLOPE) which is a mixed methodology based on model-free (MF) knockoffs and sorted Lone penalized estimation (SLOPE). mSLOPE uses original design matrix augmented with MF knockoffs matrix. The original feature matrix and MF matrix have the same covariance structure. Advantages of mSLOPE include, (i) it identifies true regressors in any dimension, (ii) it is an adaptive and computationally tractable and (iii) it gains power relative to its competitors through an exact control of FDR in any dimensions. Experimental results on both synthetic and real data show that mSLOPE gains superior power and accurate FDR control than state-of-the-art baselines.