The vision-based fruit recognition and localization system is the basis for the automatic operation of agricultural harvesting robots. Existing detection models are often constrained by high complexity and slow inference speed, which do not meet the real-time requirements of harvesting robots. Here, a method for apple object detection and localization is proposed to address the above problems. First, an improved YOLOX network is designed to detect the target region, with a multi-branch topology in the training phase and a single-branch structure in the inference phase. The spatial pyramid pooling layer (SPP) with serial structure is used to expand the receptive field of the backbone network and ensure a fixed output. Second, the RGB-D camera is used to obtain the aligned depth image and to calculate the depth value of the desired point. Finally, the three-dimensional coordinates of apple-picking points are obtained by combining two-dimensional coordinates in the RGB image and depth value. Experimental results show that the proposed method has high accuracy and real-time performance: F1 is 93%, mean average precision (mAP) is 94.09%, detection speed can reach 167.43 F/s, and the positioning errors in X, Y, and Z directions are less than 7 mm, 7 mm, and 5 mm, respectively.

The smart factory has been widely applied in manufacturing enterprises to meet dynamics in the global market. Bottleneck-based dispatching method (BDM) is a promising approach to improve the throughput of the system, which is mainly based on the current bottleneck. However, unexpected anomalies (e.g. order changes and machine failures) on shop-floor often lead to the bottleneck shifting which is hard to be tracked in traditional production shop-floor owing to the lack of real-time production data. To address the problem, a proactive task dispatching method based on future bottleneck prediction for a smart factory is proposed. Firstly, Internet of Things (IoT) technologies are applied to create a smart factory where manufacturing resources can be tracked and real-time and critical product data can be acquired to support accurate bottleneck prediction. Secondly, a bottleneck prediction method, that combines deep neural network (DNN) and time series analysis, is developed to predict future production bottleneck. Thirdly, based on the prediction, a future bottleneck-based dispatching method for throughput improvement is presented. Finally, several experiments are conducted to verify the effectiveness and availability of the proposed method.

With the wide application of advanced industrial Internet of Things (IIoT) and cyber physical system (CPS) technologies, the manufacturing resources assignment method is transformed from manual and passive mode to intelligent and active mode. However, due to the lack of real-time analysis and accurate prediction of production performance, the production adjustment demands are often released after production exceptions happen, and production decisions are often made based on historical production information, which may lead to the problem of production interruption or performance reduction. To address this issue, a proactive manufacturing resources assignment (PMRA) method based on production performance prediction for the smart factory is proposed. First, the advanced IIoT and CPS technologies are applied to create a cloud-edge cooperation environment for a smart factory, where the resources are made smart with distributed control capacity, and cloud center and edge resources can collaborate dynamically. Second, a real-time colored Petri net enabled key production performance indicators analysis and prediction method are proposed to extract real-time production information and predict future production status accurately. Then, the PMRA method is presented to assign the resources before production exceptions happen. Finally, a case study from a typical manufacturer for computer numerical control machine tools in North China is used to validate the proposed method and results show that the proposed PMRA method can largely reduce the total tardiness and the total energy consumption.

The rapid development of artificial intelligence technologies has promoted the emergence of Agriculture 4.0, where the machines participating in agricultural activities are made smart with the capacities of self-sensing, self-decision-making, and self-execution. As representative implementations of Agriculture 4.0, intelligent selective fruit harvesting robots demonstrate significant potential to alleviate labor-intensive demands in modern agriculture, where visual detection serves as the foundational component. However, the accurate detection of fruits remains a challenging issue due to the complex and unstructured nature of fruit orchards. This paper comprehensively reviews the recent progress in visual detection methods for selective fruit harvesting robots, covering cameras, traditional detection based on handcrafted feature methods, detection based on deep learning methods, and tree branch detection methods. Furthermore, the potential challenges and future trends of the visual detection system of selective fruit harvesting robots are critically discussed, facilitating a thorough comprehension of contemporary progress in this research area. The primary objective of this work is to highlight the pivotal role of visual perception in intelligent fruit harvesting robots.