Hussein Hazimeh

Hussein Hazimeh

Auto-generated profile, based on publicly available information

Hussein Hazimeh

MIT Operations Research Center

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 September 2020 in Operations Research
Verified
Journal Article
Closed access
HHHussein Hazimeh
Hussein Hazimeh
RMRahul Mazumder
Rahul Mazumder
In several scientific and industrial applications, it is desirable to build compact, interpretable learning models where the output depends on a small number of input features. Recent work has shown that such best-subset selection-type problems can be solved with modern mixed integer optimization solvers. Despite their promise, such solvers often come at a steep computational price when compared with open-source, efficient specialized solvers based on convex optimization and greedy heuristics. In “Fast Best-Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms,” Hussein Hazimeh and Rahul Mazumder push the frontiers of computation for best-subset-type problems. Their algorithms deliver near-optimal solutions for problems with up to a million features—in times comparable with the fast convex solvers. Their work suggests that principled optimization methods play a key role in devising tools central to interpretable machine learning, which can help in gaining a deeper understanding of their statistical properties.

18 October 2021 in Mathematical Programming
Verified
Journal Article
Closed access
PDF file
HHHussein Hazimeh
Hussein Hazimeh
RMRahul Mazumder
Rahul Mazumder
ASAli Saab
Ali Saab

4 August 2023
Verified
Conference Paper
Open access
SIShibal Ibrahim
Shibal Ibrahim
WCWenyu Chen
Wenyu Chen
HHHussein Hazimeh
Hussein Hazimeh
NPNatalia Ponomareva
Natalia Ponomareva
ZZZhe Zhao
Zhe Zhao
RMRahul Mazumder
Rahul Mazumder
The sparse Mixture-of-Experts (Sparse-MoE) framework efficiently scales up model capacity in various domains, such as natural language processing and vision. Sparse-MoEs select a subset of the "experts" (thus, only a portion of the overall network) for each input sample using a sparse, trainable gate. Existing sparse gates are prone to convergence and performance issues when training with first-order optimization methods. In this paper, we introduce two improvements to current MoE approaches. First, we propose a new sparse gate: COMET, which relies on a novel tree-based mechanism. COMET is differentiable, can exploit sparsity to speed up computation, and outperforms state-of-the-art gates. Second, due to the challenging combinatorial nature of sparse expert selection, first-order methods are typically prone to low-quality solutions. To deal with this challenge, we propose a novel, permutation-based local search method that can complement first-order methods in training any sparse gate, e.g., Hash routing, Top-k, DSelect-k, and COMET. We show that local search can help networks escape bad initializations or solutions. We performed large-scale experiments on various domains, including recommender systems, vision, and natural language processing. On standard vision and recommender systems benchmarks, COMET+ (COMET with local search) achieves up to 13% improvement in ROC AUC over popular gates, e.g., Hash routing and Top-k, and up to 9% over prior differentiable gates e.g., DSelect-k. When Top-k and Hash gates are combined with local search, we see up to 100X reduction in the budget needed for hyperparameter tuning. Moreover, for language modeling, our approach improves over the state-of-the-art MoEBERT model for distilling BERT on 5/7 GLUE benchmarks as well as SQuAD dataset.

3 August 2025
Verified
Conference Paper
Open access
MMMehdi Makni
Mehdi Makni
KBKayhan Behdin
Kayhan Behdin
GAGabriel Afriat
Gabriel Afriat
ZXZheng Xu
Zheng Xu
SVSergei Vassilvitskii
Sergei Vassilvitskii
HHHussein Hazimeh
Hussein Hazimeh

25 January 2020 in arXiv
Verified
Preprint
Open access
PDF file
HHHussein Hazimeh
Hussein Hazimeh
RMRahul Mazumder
Rahul Mazumder
The $L_0$-regularized least squares problem (a.k.a. best subsets) is central to sparse statistical learning and has attracted significant attention across the wider statistics, machine learning, and optimization communities. Recent work has shown that modern mixed integer optimization (MIO) solvers can be used to address small to moderate instances of this problem. In spite of the usefulness of $L_0$-based estimators and generic MIO solvers, there is a steep computational price to pay when compared to popular sparse learning algorithms (e.g., based on $L_1$ regularization). In this paper, we aim to push the frontiers of computation for a family of $L_0$-regularized problems with additional convex penalties. We propose a new hierarchy of necessary optimality conditions for these problems. We develop fast algorithms, based on coordinate descent and local combinatorial optimization, that are guaranteed to converge to solutions satisfying these optimality conditions. From a statistical viewpoint, an interesting story emerges. When the signal strength is high, our combinatorial optimization algorithms have an edge in challenging statistical settings. When the signal is lower, pure $L_0$ benefits from additional convex regularization. We empirically demonstrate that our family of $L_0$-based estimators can outperform the state-of-the-art sparse learning algorithms in terms of a combination of prediction, estimation, and variable selection metrics under various regimes (e.g., different signal strengths, feature correlations, number of samples and features). Our new open-source sparse learning toolkit L0Learn (available on CRAN and Github) reaches up to a three-fold speedup (with $p$ up to $10^6$) when compared to competing toolkits such as glmnet and ncvreg.

11 July 2020 in arXiv
Verified
Preprint
Open access
PDF file
HHHussein Hazimeh
Hussein Hazimeh
NPNatalia Ponomareva
Natalia Ponomareva
PMPetros Mol
Petros Mol
ZTZhenyu Tan
Zhenyu Tan
RMRahul Mazumder
Rahul Mazumder
Neural networks and tree ensembles are state-of-the-art learners, each with its unique statistical and computational advantages. We aim to combine these advantages by introducing a new layer for neural networks, composed of an ensemble of differentiable decision trees (a.k.a. soft trees). While differentiable trees demonstrate promising results in the literature, they are typically slow in training and inference as they do not support conditional computation. We mitigate this issue by introducing a new sparse activation function for sample routing, and implement true conditional computation by developing specialized forward and backward propagation algorithms that exploit sparsity. Our efficient algorithms pave the way for jointly training over deep and wide tree ensembles using first-order methods (e.g., SGD). Experiments on 23 classification datasets indicate over 10x speed-ups compared to the differentiable trees used in the literature and over 20x reduction in the number of parameters compared to gradient boosted trees, while maintaining competitive performance. Moreover, experiments on CIFAR, MNIST, and Fashion MNIST indicate that replacing dense layers in CNNs with our tree layer reduces the test loss by 7-53% and the number of parameters by 8x. We provide an open-source TensorFlow implementation with a Keras API.