Early detection of Basal Stem Rot (BSR) disease in oil palms is an important plantation management activity in Southeast Asia. Practical approaches for the best strategic approach toward the treatment of this disease that originated from Ganoderma Boninense require information about the status of infection. In spite of the availability of conventional methods to detect this disease, they are difficult to be used in plantation areas that are commonly large in terms of planting hectarage; therefore, there is an interest for a quick and delicate technique to facilitate the detection and monitoring of Ganoderma in its early stage. The main goal of this paper is to evaluate the use of remote sensing technique for the rapid detection of Ganoderma-infected oil palms using Unmanned Aerial Vehicle (UAV) imagery integrated with an Artificial Neural Network (ANN) model. Principally, we sought for the most representative mean and standard deviation values from green, red, and near-infrared bands, as well as the best palm circle radius, threshold limit, and the number of hidden neurons for different Ganoderma severity levels. With the obtained modified infrared UAV images at 0.026 m spatial resolution, early BSR infected oil palms were most satisfactorily detected with mean and standard deviation derived from a circle radius of 35 pixels of band green and near-infrared, 1/8 threshold limit, and ANN network by 219 hidden neurons, where the total classification accuracies achieved for training and testing the dataset were 97.52% and 72.73%, respectively. The results from this study signified the utilization of an affordable digital camera and UAV platforms in oil palm plantation, predominantly in disease management. The UAV images integrated with the Levenberg–Marquardt training algorithm illustrated its great potential as an aerial surveillance tool to detect early Ganoderma-infected oil palms in vast plantation areas in a rapid and inexpensive manner.