Researchers are working towards automation of ontology construction to manage the ever-growing data on the web. Currently, there is a shift from the use of machine learning techniques towards exploration of deep learning models for ontology construction. Deep learning model are capable of extracting terms, entities, relations, and classifications, and perform axiom learning from the underutilized richness of web-based knowledge. There has been remarkable progress in automatic ontology creation using deep learning models since they can perform word embedding, long-term dependency acquisition, concept extraction from large corpora, and inference of abstracted relationships based on broad corpora. Despite their emerging importance, deep learning models remain underutilized in ontology construction, and there is no comprehensive review of their application in ontology learning. This paper presents a comprehensive review of existing deep-learning models for the construction of ontologies, the strength and the weaknesses presented by the deep learning models for ontology learning as well as promising directions to achieve a more robust deep learning models. The Deep Learning models reviewed include Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Long-Short Term Memory (LSTMs), and Gated Recurrent Unit (GRU) as well as their ensembles. While these traditional deep learning models have achieved great success, one of their limitations is that they struggle to understand the meaning and order of data in sequences. CNNs and RNN-based models such as LSTMs and GRUs can be computationally expensive due to their large number of parameters or complex gating mechanisms. Furthermore, RNN models suffer from vanishing gradients, making it difficult to learn long-term relationships in sequences. Additionally, RNN-based models process information sequentially, limiting their ability to take advantage of powerful parallel computing hardware, slowing down training and inference, especially for long sequences. Consequently, there has been a shift towards Generative Pre-Trained (GPT) models and Bidirectional Encoder Representations from Transformers (BERT) models. This paper also reviewed the GPT-3, GPT-4, and the BERT models for extracting terms, entities, relations, and classifications. While GPT models excel in contextual understanding and flexibility, they fall short when handling domain-specific terminology and disambiguating complex relationships. Fine-tuning and domain-specific training data could minimize these shortcomings, and further enhance the performance of GPT in term and relation extraction tasks. On the other hand, the BERT models excel in comprehending context-heavy texts, but struggles with higher-level abstraction and inference tasks due to a lack of explicit semantic knowledge, thus necessitating inference for unspecified relationships. The paper recommends further research on deep learning models for ontology alignment and merging. Also, the ensembling of deep learning models and the use of domain-specific knowledge for ontology learning require further research for ontology construction.

Transformer models are significantly advancing the field of natural language processing (NLP), particularly in tasks such as the extraction of concepts and relationships from texts. However, while various studies employ single transformer models for one or both tasks and achieve performance improvements, there remain some limitations. A single model may struggle to optimize for both tasks simultaneously, resulting in poor performance in one or both areas. Furthermore, a single model may experience increased computational and memory demands when used for both tasks, as it must process and retain both entity-level and relation-level features. This can lead to inefficiencies and poor performance, especially on large or complex datasets. Moreover, when a single transformer model handles both tasks, errors in concept extraction can directly affect relation extraction, as the latter relies on accurate identification of entities. To address these limitations, this study introduces a novel Multimodal Transformer-based Fusion (MTF) model that integrates token-level representations and entity representations from different transformer models for enhanced relation classification. The proposed MTF model employs a fused cross-attention mechanism to effectively integrate these heterogeneous representations within the transformer encoder, leveraging complementary linguistic and entity-specific information. Through a structured tokenization process, we generate unified token and entity embeddings, enabling the model to learn a more distinctive representation space for improved relational understanding. The proposed MTF model is evaluated on five transformer-based architectures using the Conll04 and NYT benchmark datasets. The transformer-based architectures include three models for concept extration, namely, BERT, DistilBERT and RoBERTa, and two models for relation extraction, namely, LUKE and mLUKE. The comparison of individual model results reveal RoBERTa and mLuke as the best performing models for concept extraction and relation extraction, respectively. On both tasks, fusing the best two performing models results in the proposed MTF model outperforming the best individual models in terms of precision, recall, and F1 score. Furthermore, a comparative analysis of the results with related studies, shows the superior performance of the proposed MTF model with higher accuracy, precision, recall, and F1-score of 96.86%, 96.89%, 96.86%, 96.87%, respectively.

Deep learning approaches, utilizing Bidirectional Encoder Representation from Transformers (BERT) and advanced fine-tuning techniques, have achieved state-of-the-art accuracies in the domain of term extraction from texts. However, BERT presents some limitations in that it primarily captures the semantic context relative to the surrounding text without considering how relevant or central a token is to the overall document content. There has also been research on the application of sequence labeling on contextualized embeddings; however, the existing methods often rely solely on local context for extracting key phrases from texts. To address these limitations, this study proposes a centrality-weighted BERT model for key phrase extraction from text using sequence labelling (CenBERT-SEQ). The proposed CenBERT-SEQ model utilizes BERT to represent terms with various contextual embedding architectures, and introduces a centrality-weighting layer that integrates document-level context into BERT. This layer leverages document embeddings to influence the importance of each term based on its relevance to the entire document. Finally, a linear classifier layer is employed to model the dependencies between the outputs, thereby enhancing the accuracy of the CenBERT-SEQ model. The proposed CenBERT-SEQ model was evaluated against the standard BERT base-uncased model using three Computer Science article datasets, namely, SemEval-2010, WWW, and KDD. The experimental results show that, although the CenBERT-SEQ and BERT-base models achieved higher and close comparable accuracy, the proposed CenBERT-SEQ model achieved higher precision, recall, and F1-score than the BERT-base model. Furthermore, a comparison of the proposed CenBERT-SEQ model to that of related studies revealed that the proposed CenBERT-SEQ model achieved a higher accuracy, precision, recall, and F1-score of 95%, 97%, 91%, and 94%, respectively, than related studies, showing the superior capabilities of the CenBERT-SEQ model in keyphrase extraction from scientific documents.

Ontology learning from unstructured text has become a critical task for knowledge-driven applications in Big Data and Artificial Intelligence. While significant advances have been made in the automatic extraction of concepts and relations using neural and Transformer-based models, the generation of formal Description Logic axioms required for constructing logically consistent and computationally tractable ontologies remains largely underexplored. This paper puts forward a novel pipeline for automated axiom generation through schema mapping. Our paper introduces three key innovations: a deterministic mapping framework that guarantees logical consistency (unlike stochastic Large Language Models); guaranteed formal consistency verified by OWL reasoners (unaddressed by prior statistical methods); and a transparent, scalable bridge from neural extractions to symbolic logic, eliminating manual post-processing. Technically, the pipeline builds upon the outputs of a Transformer-based fusion model for joint concept and relation extraction. We then map lexical relational phrases to formal ontological properties through a lemmatization-based schema alignment step. Entity typing and hierarchical induction are then employed to infer class structures, as well as domain and range constraints. Using RDFLib and structured data processing, we transform the extracted triples into both assertional (ABox) and terminological (TBox) axioms expressed in Description Logic. Experimental evaluation on benchmark datasets (Conll04 and NYT) demonstrates the efficacy of the approach, with expert validation showing high acceptance rates (>95%) and reasoners confirming zero inconsistencies. The pipeline thus establishes a reliable, scalable foundation for automated ontology learning, advancing the field from extraction to formally verifiable knowledge base construction.