Video surveillance as an important function of Internet of Video Things (IoVT) system has been widely used in coal mine monitoring for coal mine safety with excellent results, however, there are still many shortcomings: 1) existing coal mine IoVT systems have limited detection accuracy for small-sized objects; 2) coal mine video surveillance systems generally adopt centralized cloud computing, transmission of massive data causes high latency, which seriously affects the response speed of object detection function; and 3) the concept drift caused by the data stream seriously affect the detection effect of the offline algorithm. To address the above issues, we propose a small object detection method-based federated intelligence (FI) to assist coal mine IoVT for object detection. First, we design a lightweight neural network Rep-ShuffleNet to improve YOLOv8, the state-of-the-art YOLO algorithm, to maintain high detection accuracy while dramatically increasing the inference speed, and with the advantage of lightweight, it can be deployed to embedded devices for low-latency edge computing; Moreover, we design a federated learning (FL)-based multilayer collaborative FL algorithm for local algorithms’ automatic and efficient optimization by asynchronous communication and data interaction reduction strategy. The experimental results show that with the assistance of FI model optimization strategies, the lightweight YOLOv8 has excellent detection performance (mean average precision: 94.6%, APsmall: 86.7%, and frame per second: 21.6), thus to assist coal mine IoVT to realize accurate and real-time underground small object detection.