Diffusion models have emerged as a powerful new family of deep generative models with record-breaking performance in many applications, including image synthesis, video generation, and molecule design. In this survey, we provide an overview of the rapidly expanding body of work on diffusion models, categorizing the research into three key areas: efficient sampling, improved likelihood estimation, and handling data with special structures. We also discuss the potential for combining diffusion models with other generative models for enhanced results. We further review the wide-ranging applications of diffusion models in fields spanning from computer vision, natural language processing, temporal data modeling, to interdisciplinary applications in other scientific disciplines. This survey aims to provide a contextualized, in-depth look at the state of diffusion models, identifying the key areas of focus and pointing to potential areas for further exploration. Github: https://github.com/YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy

Recurrent Neural Network (RNN) has demonstrated its outstanding ability in sequence tasks and has achieved state-of-the-art in many applications, such as industrial and medical. Echo State Network (ESN) is a simple type of RNN and has emerged in the last decade as an alternative to gradient descent training-based RNN. ESN is practical, conceptually simple, and easy to implement with a strong theoretical ground. It can avoid non-converging and computationally expensive issues in gradient descent RNN methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced deep ESN opened the way to uniting the merits of deep learning and reservoir computing. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive. Successfully applying ESNs needs some experience. Thus, we reviewed over 300 related papers and provided a systematic overview for the first time. In this paper, we categorize the related methods into classical ESN, DeepESN, and combination. Then, we analyze them from the perspective of network designs and specific applications. Finally, we discuss the challenges and opportunities by proposing open problems and future work.

Background and Objectives: Seizures and other seizure-like patterns of brain activity can harm the brain and contribute to in-hospital death, particularly when prolonged. However, experts qualified to interpret electroencephalography (EEG) data are scarce. Prior attempts to automate this task have been limited by small or inadequately labeled samples and have not convincingly demonstrated generalizable expert-level performance. There exists a critical unmet need for an automated method to classify seizures and other seizure-like events with expert-level reliability. This study was conducted to develop and validate a computer algorithm that matches the reliability and accuracy of experts in identifying seizures and seizure-like events, known as “ictal-interictal-injury-continuum” (IIIC) patterns on EEG, including seizures (SZ), lateralized and generalized periodic discharges (LPD, GPD), and lateralized and generalized rhythmic delta activity (LRDA, GRDA), and in differentiating these patterns from non-IIIC patterns. Methods: We used 6,095 scalp EEGs from 2,711 patients with and without IIIC events to train a deep neural network, SPaRCNet, to perform IIIC event classification. Independent training and test datasets were generated from 50,697 EEG segments, independently annotated by 20 fellowship-trained neurophysiologists. We assessed whether SPaRCNet performs at or above the sensitivity, specificity, precision, and calibration of fellowship-trained neurophysiologists for identifying IIIC events. Statistical performance was assessed via the calibration index, and by the percentage of experts whose operating points were below the model’s receiver operating characteristic curves (ROC) and precision recall curves (PRC) for the 6 pattern classes. Results: SPaRCNet matches or exceeds most experts in classifying IIIC events based on both calibration and discrimination metrics. For SZ, LPD, GPD, LRDA, GRDA, and “Other” classes, SPaRCNet exceeds the following percentages of 20 experts – ROC: 45%, 20%, 50%, 75%, 55%, 40%; PRC: 50%, 35%, 50%, 90%, 70%, 45%; and calibration: 95%, 100%, 95%, 100%, 100%, 80%, respectively. Discussion: SPaRCNet is the first algorithm to match expert performance in detecting seizures and other seizure-like events in a representative sample of EEGs. With further development, SPaRCNet may thus be a valuable tool for expedited review of EEGs.

Since labeled samples are typically scarce in real-world scenarios, self-supervised representation learning in time series is critical. Existing approaches mainly employ the contrastive learning framework, which automatically learns to understand similar and dissimilar data pairs. However, they are constrained by the request for cumbersome sampling policies and prior knowledge of constructing pairs. Also, few works have focused on effectively modeling temporal-spectral correlations to improve the capacity of representations. In this article, we propose the cross reconstruction transformer (CRT) to solve the aforementioned issues. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we obtain the frequency domain of the time series via the fast Fourier transform (FFT) and randomly drop certain patches in both time and frequency domains. Dropping is employed to maximally preserve the global context while masking leads to the distribution shift. Then a Transformer architecture is utilized to adequately discover the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, which is called Dropped Temporal-Spectral Modeling. To discriminate the representations in global latent space, we propose instance discrimination constraint (IDC) to reduce the mutual information between different time series samples and sharpen the decision boundaries. Additionally, a specified curriculum learning (CL) strategy is employed to improve the robustness during the pretraining phase, which progressively increases the dropping ratio in the training process. We conduct extensive experiments to evaluate the effectiveness of the proposed method on multiple real-world datasets. Results show that CRT consistently achieves the best performance over existing methods by 2%–9%. The code is publicly available at https://github.com/BobZwr/Cross-Reconstruction-Transformer .

Time-series forecasting is one of the most active research topics in artificial intelligence. It has the power to bring light to problems in several areas of knowledge, such as epidemiological studies, healthcare inference, and climate change analysis. Applications in real-world time series should consider two factors for achieving reliable predictions: modeling dynamic dependencies among multiple variables and adjusting the model's intrinsic hyperparameters. An open gap in the literature is that statistical and ensemble learning approaches systematically present lower predictive performance than deep learning methods. The existing applications consistently disregard the data sequence aspect entangled with multivariate data represented in more than one time series. Conversely, this work presents a novel neural network architecture for time-series forecasting that combines the power of graph evolution with deep recurrent learning on distinct data distributions, named after Recurrent Graph Evolution Neural Network ( ReGENN ). The idea is to infer multiple multivariate relationships between co-occurring time-series by assuming that the temporal data depends not only on inner variables and intra-temporal relationships (i.e., observations from itself) but also on outer variables and inter-temporal relationships (i.e., observations from other-selves). An extensive set of experiments was conducted comparing ReGENN with tens of ensemble methods and classical statistical ones. The results outperformed both statistical and ensemble-learning approaches, showing an improvement of 64.87 percent over the competing algorithms on the SARS-CoV-2 dataset of the renowned John Hopkins University for 188 countries simultaneously. For further validation, we tested our architecture in two other public datasets of different domains, the PhysioNet Computing in Cardiology Challenge 2012 and Brazilian Weather datasets. We also analyzed the Evolution Weights arising from the hidden layers of ReGENN to describe how the variables of the dataset interact with each other; and, as a result of looking at inter and intra-temporal relationships simultaneously, we concluded that time-series forecasting is majorly improved if paying attention to how multiple multivariate data synchronously evolve.