A sentence embedding vector can be obtained by connecting a global average pooling (GAP) to a pre-trained language model. The problem of such a sentence embedding vector using a GAP is that it is generated with the same weight for all words appearing in the sentence. We propose a novel sentence embedding-method-based model Token Attention-SentenceBERT (TA-SBERT) to address this problem. The rationale of TA-SBERT is to enhance the performance of sentence embedding by introducing three strategies. First, we convert the base form while preprocessing the input sentence to reduce misunderstanding. Second, we propose a novel Token Attention (TA) technique that distinguishes important words to produce more informative sentence vectors. Third, we increase stability of fine-tuning to avoid catastrophic forgetting by adding a reconstruction loss to the word embedding vector. Extensive ablation studies demonstrate that our TA-SBERT outperforms the original SentenceBERT (SBERT) in the sentence vector evaluation using semantic textual similarity (STS) tasks and the SentEval toolkit.

Many real-world applications require precise and fast time-series forecasting. Recent trends in time-series forecasting models are shifting from LSTM-based models to Transformer-based models. However, the Transformer-based model has a limited ability to represent sequential relationships in time-series data. In addition, the transformer-based model suffers from slow training and inference speed due to the bottleneck incurred by a deep encoder and step-by-step decoder inference. To address these problems, we propose a time-series forecasting optimized Transformer model, called TS-Fastformer. TS-Fastformer introduces three new optimizations: First, we propose a Sub Window Tokenizer for compressing input in a simple manner. The Sub Window Tokenizer reduces the length of input sequences to mitigate the complexity of self-attention and enables both single and multi-sequence learning. Second, we propose Time-series Pre-trained Encoder to extract effective representations through pre-training. This optimization enables TS-Fastformer to capture both seasonal and trend representations as well as to mitigate bottlenecks of conventional transformer models. Third, we propose the Past Attention Decoder to forecast target by incorporating past long short-term dependency patterns. Furthermore, Past Attention Decoder achieves high performance improvement by removing a trend distribution that changes over a long period. We evaluate the efficiency of our model with extensive experiments using seven real-world datasets and compare our model to six representative time-series forecasting approaches. The results show that the proposed TS-Fastformer reduces MSE by 10.1% compared to state-of-the-art model and demonstrates 21.6% faster training time compared to the existing fastest transformer, respectively.

As the relative importance of renewable energy in electric power systems increases, the prediction of photovoltaic (PV) power generation has become a crucial technology, for improving stability in the operation of next-generation power systems, such as microgrid and virtual power plants (VPP). In order to improve the accuracy of PV power generation forecasting, a fair amount of research has been applied to weather forecast data (to a learning process). Despite these efforts, the problems of forecasting PV power generation remains challenging since existing methods show limited accuracy due to inappropriate cloud amount forecast data, which are strongly correlated with PV power generation. To address this problem, we propose a PV power forecasting model, including a cloud amount forecasting network trained with satellite images. In addition, our proposed model adopts convolutional self-attention to effectively capture historical features, and thus acquire helpful information from weather forecasts. To show the efficacy of the proposed cloud amount forecast network, we conduct extensive experiments on PV power generation forecasting with and without the cloud amount forecast network. The experimental results show that the Mean Absolute Percentage Error (MAPE) of our proposed prediction model, combined with the cloud amount forecast network, are reduced by 22.5% compared to the model without the cloud amount forecast network.

A well-known limitation of existing rule-based text augmentation is that it cannot be applied to other languages because it depends on grammatical and structural characteristics. Moreover, most text Generative Adversarial Networks (GAN) are unstable in training due to inefficient generator optimization and rely on maximum likelihood pre-training. This paper addresses the above problems by proposing a novel augmentation method with a Sentence Generator (SG) and Sentence Discriminator (SD) for Iterative Translation-based Data Augmentation (ITDA). This paper makes three original contributions. First, the ITDA SG is designed to provide universal multiple-language support by generating comprehensive augmented sentences through serial and parallel iterations of an existing translator, such as Google Translate. Second, given that the quality of the generated sentences varies depending on the translation combination or the type of sentence, the ITDA addresses this issue using a discriminator to achieve sentence augmentation, which can select high-quality augmented data using a text classifier. Third, the ITDA can perform sentence augmentation for 109 different languages using discriminators based on text classifiers trained for a specific language or type of data set. Extensive experiments are conducted to evaluate the efficacy of the ITDA using a Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (BiLSTM), CNN-BiLSTM, and self-attention. The results demonstrate that when the ITDA is applied to 480 sentence classification tasks, the average accuracy increases by 4.24%.

Background Acute dyspnoea is common in acute care settings. However, identifying the origin of dyspnoea in the emergency department (ED) is often challenging. We aimed to investigate whether our artificial intelligence (AI)-powered ECG analysis reliably distinguishes between the causes of dyspnoea and evaluate its potential as a clinical triage tool for comparing conventional heart failure diagnostic processes using natriuretic peptides. Methods A retrospective analysis was conducted using an AI-based ECG algorithm on patients ≥18 years old presenting with dyspnoea at the ED from February 2006 to September 2023. Patients were categorised into cardiac or pulmonary origin groups based on initial admission. The performance of an AI-ECG using a transformer neural network algorithm was assessed to analyse standard 12-lead ECGs for accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUC). Additionally, we compared the diagnostic efficacy of AI-ECG models with N-terminal probrain natriuretic peptide (NT-proBNP) levels to identify cardiac origins. Results Among the 3105 patients included in the study, 1197 had cardiac-origin dyspnoea. The AI-ECG model demonstrated an AUC of 0.938 and 88.1% accuracy for cardiac-origin dyspnoea. The sensitivity, specificity and positive and negative predictive values were 93.0%, 79.5%, 89.0% and 86.4%, respectively. The F1 score was 0.828. AI-ECG demonstrated superior diagnostic performance in identifying cardiac-origin dyspnoea compared with NT-proBNP. True cardiac origin was confirmed in 96 patients in a sensitivity analysis of 129 patients with a high probability of cardiac origin initially misdiagnosed as pulmonary origin predicted by AI-ECG. Conclusions AI-ECG demonstrated superior diagnostic accuracy over NT-proBNP and showed promise as a clinical triage tool. It is a potentially valuable tool for identifying the origin of dyspnoea in emergency settings and supporting decision-making.