Music carries emotional information and allows the listener to feel the emotions contained in the music. This study proposes a multimodal music emotion recognition (MER) system using Indonesian song and lyrics data. In the proposed multimodal system, the audio data will use the mel spectrogram feature, and the lyrics feature will be extracted by going through the tokenizing process from XLNet. Convolutional long short term memory network (CNN-LSTM) performs the audio classification task, while XLNet transformers performs the lyrics classification task. The outputs of the two classification tasks are probability weight and actual prediction with the value of positive, neutral, and negative emotions, which are then combined using the stacking ensemble method. The combined output will be trained into an artificial neural network (ANN) model to get the best probability weight output. The multimodal system achieves the best performance with an accuracy of 80.56%. The results showed that the multimodal method of recognizing musical emotions gave better performance than the single modal method. In addition, hyperparameter tuning can affect the performance of multimodal systems.
Sign Language Recognition (SLR) is a relatively challenging research field which allows opportunity for improvements. In this research, we propose sentence-level SLR using deep learning method by combining Convolutional Neural Network (CNN) and Bidirectional Recurrent Neural Network (Bi-RNN). Specifically, 3D CNN is implemented to extract features from each video frame and bidirectional-RNN is implemented to extract the unique features from the video frame's sequential behavior, which later generate a possible sentence. There are two key takeaways from this paper. The first is our proposed dataset of Indonesian Sign Language (SIBI) which is comprised of 30 sentences in SIBI. The second is our novel approach of using deep learning and Connectionist Temporal Classification (CTC) loss function in sentence-level SLR. The result shows that the hyperparameter used, in this case Hyperparameter 1, achieves the best result. Also, this research found that deeper network does not necessarily guarantee good results. A bigger number of dataset also affects the performance of the system.
1 December 2022 in Bulletin of Electrical Engineering and Informatics
Language identification is at the forefront of assistance in many applications, including multilingual speech systems, spoken language translation, multilingual speech recognition, and human-machine interaction via voice. The identification of indonesian local languages using spoken language identification technology has enormous potential to advance tourism potential and digital content in Indonesia. The goal of this study is to identify four Indonesian local languages: Javanese, Sundanese, Minangkabau, and Buginese, utilizing deep learning classification techniques such as artificial neural network (ANN), convolutional neural network (CNN), and long-term short memory (LSTM). The selected extraction feature for audio data extraction employs mel-frequency cepstral coefficient (MFCC). The results showed that the LSTM model had the highest accuracy for each speech duration (3 s, 10 s, and 30 s), followed by the CNN and ANN models.
1 January 2021 in International Journal of Advanced Computer Science and Applications
Emotion recognition is one of the widely studied topics in speech technology. Emotions that come from speech can contain useful information for many purposes. The main aspects in speech emotion recognition are speech features, speech corpus, and machine learning algorithms as the classifier method. In this paper, cross-corpus method is used to conduct Indonesian Speech Emotion Recognition (SER) along with the combination of Mel Frequency Cepstral Coefficients (MFCC) and Teager Energy features. Using Support Vector Machine (SVM) as classifier, the experiment result shows that applying cross-corpus method by adding corpora from other languages to the training dataset improves the emotion classification accuracy by 4.16% on MFCC Statistics feature and 2.09% on Teager-MFCC Statistics feature.
1 February 2025 in Bulletin of Electrical Engineering and Informatics
Email phishing is a manipulative technique aimed at compromising information security and user privacy. To overcome the limitations of traditional detection methods, such as blacklists, this research proposes a phishing detection model that leverages natural language processing (NLP) and deep learning technologies to analyze Indonesian email headers. The primary objective is to more efficiently detect zero-day phishing attacks by focusing on the unique linguistic and cultural context of the Indonesian language. This enables the development of models capable of recognizing phishing attack patterns that differ from those in other language contexts. Four models are tested, combining Indonesian bidirectional encoder representation of transformers (IndoBERT) and FastText feature extraction techniques with convolutional neural network (CNN) and long short-term memory (LSTM) deep learning algorithms. The results indicate that the combination of FastText and CNN achieved the highest performance in accuracy, precision, and F1-score metrics, each at 98.4375%. Meanwhile, the FastText model with LSTM showed the best performance in recall, with a score of 98.9583%. The research suggests exploring deeper into email content or integrating analysis between headers and email content in future studies to further improve accuracy and effectiveness in phishing email detection.
1 October 2023 in Bulletin of Electrical Engineering and Informatics
Humans’ fundamental need is interaction with each other such as using conversation or speech. Therefore, it is crucial to analyze speech using computer technology to determine emotions. The speech emotion recognition (SER) method detects emotions in speech by examining various aspects. SER is a supervised method to decide the emotion class in speech. This research proposed a multimodal SER model using one of the deep learning based enhancement techniques, which is the attention mechanism. Additionally, this research addresses the imbalanced dataset problem in the SER field using generative adversarial networks (GAN) as a data augmentation technique. The proposed model achieved an excellent evaluation performance of 0.96 or 96% for the proposed GAN configuration. This work showed that the GAN method in the multimodal SER model could enhance performance and create a balanced dataset.