Multiarmed bandit (MAB) is a classic model for capturing the exploration–exploitation trade-off inherent in many sequential decision-making problems. The classic MAB framework, however, only allows “local” constraints on decisions and “sum of rewards” as objective. In many real-world applications, there are multiple complex constraints on resources that are consumed during the entire decision process, and performance may be evaluated through nonlinear utility functions on aggregate rewards. This article presents a new MAB framework that allows such “global” convex constraints and concave objective functions along with new algorithmic techniques with provably near-optimal performance bounds. The authors discuss applications in several domains, such as network revenue management, crowdsourcing, and pay-per-click advertising, which benefit from the new more general framework by admitting richer models and more efficient risk-averse solutions.

We present an algorithm based on posterior sampling (aka Thompson sampling) that achieves near-optimal worst-case regret bounds when the underlying Markov decision process (MDP) is communicating with a finite, although unknown, diameter. Our main result is a high probability regret upper bound of $\tilde{O}(DS\sqrt{AT})$ for any communicating MDP with S states, A actions, and diameter D. Here, regret compares the total reward achieved by the algorithm to the total expected reward of an optimal infinite-horizon undiscounted average reward policy in time horizon T. This result closely matches the known lower bound of $\mathrm{\Omega }(\sqrt{\mathit{DSAT}})$ . Our techniques involve proving some novel results about the anti-concentration of Dirichlet distribution, which may be of independent interest.

Albuminuria is both a hallmark and a risk factor for progressive glomerular disease, and results in increased exposure of podocytes to serum albumin with its associated factors. Here in vivo and in vitro models of serum albumin-overload were used to test the hypothesis that albumin-induced proteinuria and podocyte injury directly correlate with COX-2 induction. Albumin induced COX-2, MCP-1, CXCL1, and the stress protein HSP25 in both rat glomeruli and cultured podocytes, whereas B7-1 and HSP70i were also induced in podocytes. Podocyte exposure to albumin induced both mRNA and protein and enhanced the mRNA stability of COX-2, a key regulator of renal hemodynamics and inflammation, which renders podocytes susceptible to injury. Podocyte exposure to albumin also stimulated several kinases (p38 MAPK, MK2, JNK/SAPK, and ERK1/2), inhibitors of which (except JNK/SAPK) downregulated albumin-induced COX-2. Inhibition of AMPK, PKC, and NFκB also downregulated albumin-induced COX-2. Critically, albumin-induced COX-2 was also inhibited by glucocorticoids and thiazolidinediones, both of which directly protect podocytes against injury. Furthermore, specific albumin-associated fatty acids were identified as important contributors to COX-2 induction, podocyte injury, and proteinuria. Thus, COX-2 is associated with podocyte injury during albuminuria, as well as with the known podocyte protection imparted by glucocorticoids and thiazolidinediones. Moreover, COX-2 induction, podocyte damage, and albuminuria appear mediated largely by serum albumin-associated fatty acids.

A fundamental yet notoriously difficult problem in operations management is the periodic inventory control problem under positive lead time and lost sales. More recently, there has been interest in the problem setting where the demand distribution is not known a priori and must be learned from the observations made during the decision-making process. In “Learning in Structured MDPs with Convex Cost Functions: Improved Regret Bounds for Inventory Management,” Agrawal and Jia present a reinforcement learning algorithm that uses the observed outcomes of past decisions to implicitly learn the underlying dynamics and adaptively improve the decision-making strategy over time. They show that, compared with the best base-stock policy, their algorithm achieves an optimal regret bound in terms of the time horizon and scales linearly with the lead time of the inventory ordering process. Furthermore, they demonstrate that their approach is not restricted to the inventory problem and can be applied in an almost black box manner to more general reinforcement learning problems with convex cost functions.

Nephrotic syndrome (NS) is a characterized by massive proteinuria, edema, hypoalbuminemia, and dyslipidemia. Glucocorticoids (GCs), the primary therapy for >60 years, are ineffective in approximately 50% of adults and approximately 20% of children. Unfortunately, there are no validated biomarkers able to predict steroid-resistant NS (SRNS) or to define the pathways regulating SRNS. We performed proteomic analyses on paired pediatric NS patient plasma samples obtained both at disease presentation before glucocorticoid initiation and after approximately 7 weeks of GC therapy to identify candidate biomarkers able to either predict steroid resistance before treatment or define critical molecular pathways/targets regulating steroid resistance. Proteomic analyses of 15 paired NS patient samples identified 215 prevalent proteins, including 13 candidate biomarkers that predicted SRNS before GC treatment, and 66 candidate biomarkers that mechanistically differentiated steroid-sensitive NS (SSNS) from SRNS. Ingenuity Pathway Analyses and protein networking pathways approaches further identified proteins and pathways associated with SRNS. Validation using 37 NS patient samples (24 SSNS/13 SRNS) confirmed vitamin D binding protein (VDB) and APOL1 as strong predictive candidate biomarkers for SRNS, and VDB, hemopexin (HPX), adiponectin (ADIPOQ), sex hormone-binding globulin (SHBG), and APOL1 as strong candidate biomarkers to mechanistically distinguish SRNS from SSNS. Logistic regression analysis identified a candidate biomarker panel (VDB, ADIPOQ, and matrix metalloproteinase 2 [MMP-2]) with significant ability to predict SRNS at disease presentation (P = 0.003; area under the receiver operating characteristic curve = 0.78). Plasma proteomic analyses and immunoblotting of serial samples in childhood NS identified a candidate biomarker panel able to predict SRNS at disease presentation, as well as candidate molecular targets/pathways associated with clinical steroid resistance.