Wildland fires, a natural calamity, pose a significant threat to both human lives and the environment while causing extensive economic damage. As the use of Unmanned Aerial Vehicles (UAVs) with computer vision in disaster management continues to grow, there is a rising need for effective wildfire classification and localization. We propose a multi-stream hybrid deep learning model with a dual-stream attention mechanism for classifying wildfires from aerial and territorial images. Our proposed method incorporates a pre-trained EfficientNetB7 and customized Attention Connected Network (ACNet). This approach demonstrates exceptional classification performance on two widely recognized benchmark datasets. Bayesian optimization is employed for the purpose of refining and optimizing the hyperparameters of the model. The proposed model attains 97.45%, 98.20%, 97.10%, and 97.12% as accuracy, precision, recall, and F1-score, respectively, on the FLAME dataset. Moreover, while evaluated on the DeepFire dataset, the model achieves accuracy, precision, recall, and F1-scores of 95.97%, 95.19%, 96.01%, and 95.54%, respectively. The proposed method achieved a TNR of 95.5% and a TPR of 99.3% on the FLAME dataset, as well as a TNR of 94.47% and a TPR of 96.82% on the DeepFire dataset. This performance surpasses numerous state-of-the-art methods. To demonstrate the interpretability of our model, we incorporated the GRAD-CAM technique, which enables us to precisely identify the fire location within the feature map. This finding illustrates the efficacy of the model in accurately categorizing wildfires, even in areas with less fire activity.