Internet of Things networks evolve as a rapidly growing field for security threats, such as Denial-of-Service cross-layer attacks, due to their heterogeneous and resource-constrained environment. Intrusion detection systems (IDSs) serve as a vital defense mechanism in modern cybersecurity. However, the adoption of such a system, especially one that adopts a cross-layer strategy, requires a standardized, multifaceted evaluation framework that accounts for both detection capability and operational overhead. To address these challenges, we proposed a modular weight-based framework that evaluates cross-layer Machine Learning (ML) IDS across multiple dimensions, namely, detection effectiveness and generalizability, data quality, and attack coverage and practical deployability. We then applied this framework to the state-of-the-art cross-layer ML IDSs identified through the PRISMA framework. This proof-of-concept application illustrates how current evaluation practices generate disparate, fragmented results, while also highlighting the limitations inherent in retrospective literature-based scoring.