Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Large language model (LLM)-based AI coaches show promise for personalized exercise and health interventions. Their complex capabilities (communication, planning, movement analysis, monitoring) necessitate rigorous, multidimensional evaluation, but standardized frameworks are lacking. Objective: This scoping review systematically maps current evaluation strategies for LLM-based AI coaches in exercise and health, identifies strengths and limitations, and proposes future directions for robust, standardized validation. Methods: Following PRISMA-ScR guidelines, we systematically searched six databases using keywords for LLMs, exercise/health coaching, and evaluation. Studies describing LLM-based coaching systems with reported performance evaluation methods were included. Data on models, applications, evaluation strategies, and outcomes were charted. Results: Seventeen studies published between March 2023 and March 2025 met the inclusion criteria. Most utilized proprietary models (e.g., GPT-4), while some used open-source or custom models. Six studies incorporated multimodal inputs (video, sensor data). Evaluation strategies were highly heterogeneous, including quantitative metrics (Accuracy, F1, MAE), empirical methods (user studies, expert comparisons), and expert/user-centered feedback (expert scores [Kappa ≈ 0.79–0.82], user surveys [MITI, SASSI]). However, evaluations often lacked real-world testing, longitudinal assessment, and standardized benchmarks. Conclusions: Evaluating LLM-based exercise and health coaches requires multifaceted strategies: quantitative metrics for objective tasks, empirical validation for user interaction, and expert assessment for personalization and safety. Current evaluations are fragmented, lacking standardization, ecological validity, and longitudinal assessment. Future progress demands robust, multidimensional frameworks emphasizing real-world validation, integrating RAG for accuracy, and developing specialized, efficient multimodal models or agents for reliable and scalable AI coaching.