In the last few years there has been a substantial increase in the success of deep learning, especially with regard to convolutional neural networks for computer vision tasks. These architectures are being widely used in emergency situations, where a fast and accurate response is needed. In environmental monitoring, several works have focused on fire detection, since fires have been increasingly associated with negative consequences such as respiratory diseases, economical losses and the destruction of natural resources. The automatic detection of smoke and fire, however, poses a particularly difficult challenge to computer vision systems, since the variability in the shape, color and texture of these objects makes the process of learning how to detect them much more complicated than for other ordinary objects. As a consequence, the number of false positives may grow high, which is especially problematic for a real-time application that mobilizes human efforts to fight fire. This work presents a robust fire detection tool based on a 2D deep convolutional network capable of suppressing false alarms from clouds, fogs, car lights and other objects that are easily confused with fire and smoke. Our approach integrates an object detector with an object tracker; this makes it possible to analyze the temporal behavior of the object and use that information in the decision process. We also present D-Fire, a public and labeled dataset containing more than 21,000 images, which is used to train and test the proposed system. The experimental results show that the detector reached an mAP@0.50 = 75.91% and that the incorporation of the temporal context resulted in a 60% reduction in the false positive rate at the cost of a 2.86% reduction in true positive rate. In addition, the computational cost added by the proposed approach to the fire detector is negligible, so that real-time detection is still completely feasible.