Predicting host phenotypes from gut microbiome data is central to developing microbiome-based diagnostics and monitoring tools. Although many computational methods have been proposed, researchers frequently rely on traditional models such as Random Forest or Logistic Regression. This hesitancy to adopt newer methods stems from their complexity as well as the lack of standardized evaluations, as most tools are assessed on different datasets and compared against a limited set of methods. To address these challenges, we developed LAMPP — Live Assessment of Metagenomics-based tools for host Phenotype Prediction. LAMPP consists of multiple tasks covering five diverse phenotypes: inflammatory bowel disease, colorectal cancer, schizophrenia, delivery mode, and general health status. These tasks were designed to reflect real-world variation in dataset characteristics, including cohort size, class balance, technical heterogeneity, and train-test similarity (from within-cohort to cross-cohort scenarios). We evaluated a range of traditional and deep learning models, including Random Forest, XGBoost, Logistic Regression, SIAMCAT, DeepMicro, DEBIAS-M, TabPFN and fully connected neural networks. These models differ in their approaches to feature selection, normalization, and representation. Performance was measured using ROC AUC, and we additionally assessed runtime, resource usage, and usability. Our results show that prediction difficulty varies across tasks. Importantly, deep learning methods did not outperform traditional models. SIAMCAT, Random Forest, and XGBoost performed comparably, highlighting that accessible, established tools remain effective across diverse microbiome prediction scenarios. LAMPP and its leaderboard are available at https://lampp.yassourlab.com, providing a transparent platform for benchmarking emerging methods. By enabling consistent and reproducible comparisons, LAMPP fosters progress in microbiome-based phenotype prediction and supports the broader adoption of robust, accessible computational tools in microbiome research.