Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.

For dual three-phase interior permanent magnet synchronous machines (IPMSMs), open-phase fault can result in significant average torque reduction and harmonics in the output torque and speed, which prevent the machines from reliable and safe operation. Indeed, these adverse effects are mainly due to the significant harmonics in the stator currents caused by the open-phase fault. This paper investigates fault tolerant control (FTC) of dual three-phase IPMSM under open-phase fault and proposes a computation-efficient FTC solution to maximize the average torque and minimize the fault-induced torque and speed ripples. In the proposed FTC, open-phase model is firstly derived, and optimal stator currents are then derived to achieve maximized average torque and minimized fault-induced torque harmonics. The computation efficiency enables the proposed solution the capability of FTC under both the steady-state and transient conditions. Moreover, the proposed FTC can eliminate the harmonic current components in the torque contributing frame and thus reduce the harmonic losses, and nonlinear inductance maps is employed to consider magnetic saturation. The proposed FTC is compared with existing methods and evaluated with experiments on a laboratory dual three-phase IPMSM under various operating conditions.

For interior permanent magnet synchronous machine (IPMSMs), maximum torque per ampere (MTPA) control aims to find the MTPA angle to maximize the control objective (the ratio of output torque to stator current). This paper proposes a novel online polynomial curve fitting technique for fast and accurate MTPA angle detection, which is motivated by the fact that the objective increases before MTPA angle and decreases after MTPA angle. This paper proposes a polynomial based objective model and identifies the polynomial parameters from a few test data for direct MTPA angle calculation. The proposed approach can avoid the time-consuming search process resulting in fast detection speed in comparison to existing search-based methods. In implementation, the current angle is set to a few test values to obtain the data for online curve fitting and MTPA angle calculation, in which there is no need of machine inductances and PM flux linkage. Moreover, the proposed polynomial model is analyzed to obtain the number of test data required for fast and accurate MTPA angle detection. The proposed approach is validated with extensive experiments and comparisons with existing methods on a laboratory IPMSM.

In surgery, the robot can automatically adjust the endoscope to ensure the stability of the surgical image and reduce the burden on the doctor. It is becoming a research hotspot of current surgical assistance. The identification and localization of endoscopic instruments tips is the key to achieve efficient “doctor–robot” collaboration. However, traditional methods have the problems of poor real-time performance and insufficient information expression. This article proposes an autonomous recognition of multiple surgical instrument tips based on arrow object bounding box (OBB)-YOLO network prediction, the proposed method not only realizes the recognition and localization of key parts of the instrument, but also uses arrows to enhance the expression of critical information such as the tip angle under the premise of meeting the real-time requirements for medical scenes. First, based on the modified YOLOv3 model, the Arrow OBB-YOLO network used to predict Arrow OBB is built, and the arrows are constrained inside the bounding box (BB) through coordinate transformation. Second, the optimization expression of arrow errors can enhance the prediction effect of the model, and the designed step-by-step training optimization method significantly accelerates the convergence speed of the model. Then, based on the Arrow OBB model, a method for generating the tracking areas is proposed. Finally, the experimental results show that the proposed method can accurately identify and locate the tips of the instrument with 36.5 fps and can improve the recognition effect of the BB when adding new information expression, which lays the foundation for “doctor–robot” collaborative surgery.

The existing deep learning-based fault prognostic methods require massive labeled condition monitoring (CM) data to train a well-generalized model. However, acquiring massive labeled CM data for real-case machines is infeasible due to time, economic costs, and safety concerns. Fortunately, we can handily obtain labeled CM data from relevant but different machines such as from accelerated degradation experiments in laboratories, which contain partially shared prognosis knowledge correlated to real-case machines. Accordingly, to bridge this practical gap, a novel Bayesian deep dual network with domain adaptation is developed to achieve transfer fault prognosis across different machines with distinct structures, measurement settings, and operating conditions. A deep convolutional neural network (DCNN)-multiple layer perceptron (MLP) dual network is first employed to extract abundant degradation representations from time series-based and time-frequency spectrum-based raw features. Then, domain adaptation regularization is imposed to relieve significant distribution discrepancy issue existing across different machines. Finally, the proposed DCNN-MLP dual network integrated with domain adaptation module is extended into Bayesian dual network through variational-inference (VI)-based method. The experimental verification demonstrates that the proposed method can accurately predict the remaining useful life percentage of testing bearings without any labeled CM data in target domain, and comparisons with other existing methods are also included.

With the emergence of Large Language Models (LLMs), artificial intelligence (AI) has experienced revolutionary advancements. Fault diagnosis and maintenance, as crucial components of industrial production, can also undergo significant technological innovations. However, as black-box models, LLMs have three main drawbacks in fault diagnosis scenarios that require deep and responsible reasoning and decision-making: 1) LLMs face constraints in acquiring real-time factual updates knowledge within fault diagnosis scenarios and in training high-precision, domain-specific models through fine-tuning; 2) The inference results of LLMs lack interpretability and traceability when dealing with multi-level logical reasoning; 3) In specific domains, LLMs often struggle with factual hallucinations. An effective method to address these issues is to integrate external knowledge sources into the reasoning processes of LLMs. Based on the complementarity between LLMs and knowledge graphs (KGs), this paper proposes a new framework called Think on Fault Diagnosis Event Knowledge Graph (To-FD-EKG), which consists of two main components: a knowledge modeling module and a knowledge reasoning module. I) The knowledge modeling module includes the proposed table-sequence encoder model for jointly extracting multiple relations and high-density event entities (TSM-JERHDE) from real industrial scenarios, constructing the Fault Diagnosis Event Knowledge Graph (FD-EKG), and creating a Virtual Fault Diagnosis Digital Twin Environment (VFD-DTEnv). This provides reliable and updatable external knowledge for LLM reasoning, thereby extending its knowledge boundaries. II) Treating LLMs as agents, knowledge reasoning module generates both thought paths and fault diagnosis actions by interactively retrieving and reasoning on the constructed VFDDTEnv while retrospectively assessing the global context. The reasoning process is constrained by the FD-EKG, generating observable and traceable reasoning paths, reducing hallucinations and enhancing the interpretability of the results. We aggregated two years of maintenance logs from an operational wind farm to construct a textual dataset for wind turbine (WT) fault diagnosis. By assessing 1,227 WT maintenance logs, the outcomes of our practical case experiment underscore the effectiveness of the proposed methodology.