Transfer learning is a technique utilized in deep learning applications to transmit learned inference to a different target domain. The approach is mainly to solve the problem of a few training datasets resulting in model overfitting, which affects model performance. The study was carried out on publications retrieved from various digital libraries such as SCOPUS, ScienceDirect, IEEE Xplore, ACM Digital Library, and Google Scholar, which formed the Primary studies. Secondary studies were retrieved from Primary articles using the backward and forward snowballing approach. Based on set inclusion and exclusion parameters, relevant publications were selected for review. The study focused on transfer learning pretrained NLP models based on the deep transformer network. BERT and GPT were the two elite pretrained models trained to classify global and local representations based on larger unlabeled text datasets through self-supervised learning. Pretrained transformer models offer numerous advantages to natural language processing models, such as knowledge transfer to downstream tasks that deal with drawbacks associated with training a model from scratch. This review gives a comprehensive view of transformer architecture, self-supervised learning and pretraining concepts in language models, and their adaptation to downstream tasks. Finally, we present future directions to further improvement in pretrained transformer-based language models.

Tuberculosis (TB) is an airborne disease with a high fatality rate that often affects people’s lungs. Early detection of the disease can guarantee a cure, but that is not the case due to the lack of experts and screening facilities. With the advancement in Artificial Intelligence (AI) techniques, publicly available clinical datasets and high computer systems, several solutions have been proposed to automate the diagnosing process based on deep learning (DL) algorithms. This study presents a DL technique based on a combined Data efficient image transformer and the Residual Network-16 (ResNet-16) model for effective TB diagnosis from X-ray images. The TBX11K dataset got utilized for this investigation. The dataset has three categories—Healthy, Sick but non-TB and TB. Relying on only healthy X-rays as the negative category in clinical settings where there are many unwell but non-TB samples can cause considerable false positives in the model prediction, hence the adoption of the sick but non-TB class. The dataset got divided into training, validation, and testing sets for the experiment. The self-attention mechanism within the transformer part of the proposed model learns crucial information and constructs the relationship between image tokens. The ResNet-16 part uses depth-wise convolution to gather local representations and reduce computing costs while increasing the diagnosing accuracy. The global average pooling is applied to the feature maps at the final convolution layer to generate heatmaps based on the class activation map to show exactly where the prediction happened. The suggested model obtained TB diagnostic accuracy, sensitivity, specificity and precision rates of 99.38%, 99.49%, 99.26% and 99.24%, respectively. Additionally, it is lightweight (6.9M) and detects more quickly (4.79ms) than the other comparative cutting-edge versions. The suggested model’s exceptional results demonstrate its effectiveness in TB detection, allowing for practical use.

Tuberculosis (TB) is an infectious disease affecting humans’ lungs and is currently ranked the 13th leading cause of death globally. Due to advancements in technology and the availability of medical datasets, automatic analysis and classification of chest X-rays (CXRs) into TB and non-TB can be a reliable alternative for early TB screening. We propose an automatic TB detection system using advanced deep learning (DL) models. A substantial part of a CXR image is dark, with no relevant information for diagnosis and potentially confusing DL models. In this work, the U-Net model extracts the region of interest from CXRs and the segmented images are fed to the DL models for feature extraction. Eight different convolutional neural networks (CNN) models are employed in our experiments, and their classification performance is compared based on three publicly available CXR datasets. The U-Net model achieves segmentation accuracy of 98.58%, intersection over union (IoU) of 93.10, and a Dice coefficient score of 96.50. Our proposed stacked ensemble algorithm performed better by achieving accuracy, sensitivity, and specificity values of 98.38%, 98.89%, and 98.70%, respectively. Experimental results confirm that segmented lung CXR images with ensemble learning produce a better result than un-segmented lung CXR images.