Oral cancer is a fatal disease and ranks seventh among the most common cancers throughout the whole globe. Oral cancer is a type of cancer that usually affects the head and neck. The current gold standard for diagnosis is histopathological investigation, however, the conventional approach is time-consuming and requires professional interpretation. Therefore, early diagnosis of Oral Squamous Cell Carcinoma (OSCC) is crucial for successful therapy, reducing the risk of mortality and morbidity, while improving the patient’s chances of survival. Thus, we employed several artificial intelligence techniques to aid clinicians or physicians, thereby significantly reducing the workload of pathologists. This study aimed to develop hybrid methodologies based on fused features to generate better results for early diagnosis of OSCC. This study employed three different strategies, each using five distinct models. The first strategy is transfer learning using the Xception, Inceptionv3, InceptionResNetV2, NASNetLarge, and DenseNet201 models. The second strategy involves using a pre-trained art of CNN for feature extraction coupled with a Support Vector Machine (SVM) for classification. In particular, features were extracted using various pre-trained models, namely Xception, Inceptionv3, InceptionResNetV2, NASNetLarge, and DenseNet201, and were subsequently applied to the SVM algorithm to evaluate the classification accuracy. The final strategy employs a cutting-edge hybrid feature fusion technique, utilizing an art-of-CNN model to extract the deep features of the aforementioned models. These deep features underwent dimensionality reduction through principal component analysis (PCA). Subsequently, low-dimensionality features are combined with shape, color, and texture features extracted using a gray-level co-occurrence matrix (GLCM), Histogram of Oriented Gradient (HOG), and Local Binary Pattern (LBP) methods. Hybrid feature fusion was incorporated into the SVM to enhance the classification performance. The proposed system achieved promising results for rapid diagnosis of OSCC using histological images. The accuracy, precision, sensitivity, specificity, F-1 score, and area under the curve (AUC) of the support vector machine (SVM) algorithm based on the hybrid feature fusion of DenseNet201 with GLCM, HOG, and LBP features were 97.00%, 96.77%, 90.90%, 98.92%, 93.74%, and 96.80%, respectively.

Oral cancer poses a formidable global health threat, demanding urgent attention to combat its devastating impact. Timely detection of oral squamous cell carcinoma (OSCC) is pivotal for successful treatment and improved survival rates. However, manual histopathological analysis, reliant on the expertise of medical practitioners, can be time-consuming and vulnerable to subjective discrepancies. To surmount these challenges and elevate diagnostic outcomes, this research explores the transformative potential of artificial intelligence (AI) in OSCC diagnosis. Three distinct methodologies, namely Gabor + CatBoost, ResNet50 + CatBoost, and Gabor+ ResNet50 + CatBoost, were implemented to leverage the power of AI. By extracting 32 low-level features from the Gabor Filter and 100,532 high-level features from the ResNet50 model, the study adopts principal component analysis (PCA) to mitigate overfitting, retaining the top 4096 components. The extracted features underwent individual classification using CatBoost, followed by concatenation and image classification. Remarkably, the third strategy, which synergized Gabor filtering with ResNet50 feature extraction, along with CatBoost classification, demonstrated the most exceptional performance. Achieving an impressive accuracy of 94.92%, 95.51% precision, 84.30% sensitivity, 95.54% specificity, 94.90% F1 score, and 94.9% AUC, these AI-based approaches herald a new era of accurate and efficient OSCC diagnosis.