With the development of information technology, Internet-of-Things (IoT) and low-altitude remote-sensing technology represented by Unmanned Aerial Vehicles (UAVs) are widely used in environmental monitoring fields. In agricultural modernization, IoT and UAV can monitor the incidence of crop diseases and pests from the ground micro and air macro perspectives, respectively. IoT technology can collect real-time weather parameters of the crop growth by means of numerous inexpensive sensor nodes. While depending on spectral camera technology, UAVs can capture the images of farmland, and these images can be utilize for analyzing the occurrence of pests and diseases of crops. In this work, we attempt to design an agriculture framework for providing profound insights into the specific relationship between the occurrence of pests/diseases and weather parameters. Firstly, considering that most farms are usually located in remote areas and far away from infrastructure, making it hard to deploy agricultural IoT devices due to limited energy supplement, a sun tracker device is designed to adjust the angle automatically between the solar panel and the sunlight for improving the energy-harvesting rate. Secondly, for resolving the problem of short flight time of UAV, a flight mode is introduced to ensure the maximum utilization of wind force and prolong the fight time. Thirdly, the images captured by UAV are transmitted to the cloud data center for analyzing the degree of damage of pests and diseases based on spectrum analysis technology. Finally, the agriculture framework is deployed in the Yangtze River Zone of China and the results demonstrate that wheat is susceptible to disease when the temperature is between 14 °C and 16 °C, and high rainfall decreases the spread of wheat powdery mildew.