The rapid growth of Internet-of-Things (IoT) deployments has substantially expanded the attack surface of modern cyber–physical systems, making accurate and computationally feasible malware detection essential for enterprise and industrial environments. This study presents a large-scale, systematic comparison of 27 machine learning (ML) and 18 deep learning (DL) models for IoT malware detection across eight major malware categories: Trojan, Botnet, Ransomware, Rootkit, Worm, Spyware, Keylogger, and Virus. A realistic dataset was constructed using 50,000 executable samples collected from the Any.Run platform, including 8000 malware instances (1000 per class) and 42,000 benign samples. Each sample was executed in a sandbox to extract detailed static and behavioral telemetry. A targeted feature-selection pipeline reduced the feature space to 47 diagnostic features spanning static properties, behavioral indicators, process/file/registry activity, debug signals, and network telemetry, yielding a compact representation suitable for malware detection in IoT settings. Experimental results demonstrate that ensemble tree-based ML models consistently dominate performance on the engineered tabular feature set as 7 of the top 10 models are ML, with CatBoost and LightGBM achieving near-ceiling accuracy and low false-positive rates. Per-malware analysis further shows that optimal model choice depends on malware behavior. CatBoost is best for Trojan/Spyware, LightGBM for Botnet, XGBoost for Worm, Extra Trees for Rootkit, and Random Forest for Keylogger, while DL models are competitive only for specific categories, with TabNet performing best for Ransomware and FT-Transformer for Virus. In addition, an end-to-end computational time analysis across all 45 models reveals a clear efficiency advantage for boosted tree ensembles relative to most DL architectures, supporting deployment feasibility on commodity CPU hardware. Overall, the study provides actionable guidance for designing adaptive IoT malware detection frameworks, recommending gradient-boosted ensemble ML models as the primary deployment choice, with selective DL models only when category-specific gains justify additional computational cost.

This study introduces a modular, behaviorally curated malware dataset suite consisting of eight independent sets, each specifically designed to represent a single malware class: Trojan, Mirai (botnet), ransomware, rootkit, worm, spyware, keylogger, and virus. In contrast to earlier approaches that aggregate all malware into large, monolithic collections, this work emphasizes the selection of features unique to each malware type. Feature selection was guided by established domain knowledge and detailed behavioral telemetry obtained through sandbox execution and a subsequent report analysis on the AnyRun platform. The datasets were compiled from two primary sources: (i) the AnyRun platform, which hosts more than two million samples and provides controlled, instrumented sandbox execution for malware, and (ii) publicly available GitHub repositories. To ensure data integrity and prevent cross-contamination of behavioral logs, each sample was executed in complete isolation, allowing for the precise capture of both static attributes and dynamic runtime behavior. Feature construction was informed by operational signatures characteristic of each malware category, ensuring that the datasets accurately represent the tactics, techniques, and procedures distinguishing one class from another. This targeted design enabled the identification of subtle but significant behavioral markers that are frequently overlooked in aggregated datasets. Each dataset was balanced to include benign, suspicious, and malicious samples, thereby supporting the training and evaluation of machine learning models while minimizing bias from disproportionate class representation. Across the full suite, 10,000 samples and 171 carefully curated features were included. This constitutes one of the first dataset collections intentionally developed to capture the behavioral diversity of multiple malware categories within the context of Internet of Things (IoT) security, representing a deliberate effort to bridge the gap between generalized malware corpora and class-specific behavioral modeling.

The growth of Internet of Things (IoT) environments has expanded the attack surface of modern systems. Trojan attacks are a major challenge as they evade conventional detection mechanisms and operate silently within legitimate processes. This paper presents an automated Trojan detection framework for Windows-based IoT gateways. The framework combines custom dataset generation informative feature engineering and deep learning-driven analysis. A dataset of 3000 real world executable samples was created through controlled sandbox execution and forensic monitoring. The process captured behavioral static and network-level characteristics. An initial set contained 146 extracted features. A multi-stage feature selection process identified 33 informative attributes. This step allowed efficient learning and preserved discriminative power. A custom deep neural network model named TrDNN was developed using these features. The model captures complex nonlinear patterns linked to Trojan activity. The framework was evaluated against five classical machine learning models. It was also compared with five deep learning baselines. Results show that TrDNN achieves strong detection performance. The accuracy is 0.975. The precision is 0.972. The recall is 0.969. The F1 score is 0.970. The study also examines inference time and energy consumption. The model shows a balance between detection effectiveness, computational cost and energy efficiency. This makes it suitable for resource-constrained IoT gateway deployment. The detection model was integrated into an automated real-time monitoring pipeline. The system enables continuous process surveillance through Windows command line automation with minimal operational overhead. Statistical validation used paired t tests, Wilcoxon signed rank tests and McNemar chi-square test. The performance gains are statistically significant and do not indicate overfitting. The framework provides a reliable, efficient and deployable solution for Trojan detection in modern IoT systems.