Forest fires occur frequently around the world due to extreme weather conditions of high temperatures and drought. Vision-based convolutional neural networks (CNNs) have greatly improved forest fire detection accuracy. However, slow inference speed severely restricts real-time deployment in actual forest scenes. Existing models generally adopt group convolution (GConv) or depthwise convolution (DWConv) to reduce computational complexity, which causes frequent memory access and result in a practical inference speed far below theoretical expectations. Therefore, we propose a novel fast neural network named FasterNetFire for forest fire detection, which introduces partial convolution (PConv) as the basic feature extraction operator to reduce redundant computation as well as memory access overhead simultaneously. FasterNetFire is composed of four cascaded stages, each stage contains several stacked FasterNet Blocks, and the core of each FasterNet Block is an inverted residual module built upon PConv. The proposed network significantly improves inference efficiency while maintaining the effectiveness of spatial feature extraction. Experiments conducted on the FD and Foggia’s fire detection dataset demonstrate that our FasterNetFire achieves an impressive inference speed of up to 290 frames per second (FPS) on graphics processing unit (GPU) platforms. Compared with current representative methods, its inference speed is 4.5× and 4× faster than that of EFDNet and DFAN. Furthermore, FasterNetFire achieves the best results among 17 state-of-the-art methods, achieving an excellent balance between detection accuracy and real-time response performance. This advantage fully verifies the high efficiency of PConv in vision forest fire detection tasks and provides a novel lightweight solution for real-time monitoring and early warning of forest fires in resource-constrained environments.