The system complexity that characterizes current systems warrants an integrated and comprehensive approach to system design and development. This need has brought about a paradigm shift towards Model-Based Systems Engineering (MBSE) approaches to system design and a departure from traditional document-centric methods. While MBSE shows great promise, the ambiguities and inconsistencies present in Natural Language (NL) requirements hinder their conversion to models directly. The field of Natural Language Processing (NLP) has demonstrated great potential in facilitating the conversion of NL requirements into a semi-machine-readable format that enables their standardization and use in a model-based environment. A first step towards standardizing requirements consists of classifying them according to the type (design, functional, performance, etc.) they represent. To that end, a language model capable of classifying requirements needs to be fine-tuned on labeled aerospace requirements. This paper presents an open-source, annotated aerospace requirements corpus (the first of its kind) developed for the purpose of this effort that includes three types of requirements, namely design, functional, and performance requirements. This paper further describes the use of the aforementioned corpus to fine-tune BERT to obtain the aeroBERT-Classifier: a new language model for classifying aerospace requirements into design, functional, or performance requirements. Finally, this paper provides a comparison between aeroBERT-Classifier and other text classification models such as GPT-2, Bidirectional Long Short-Term Memory (Bi-LSTM), and bart-large-mnli. In particular, it shows the superior performance of aeroBERT-Classifier on classifying aerospace requirements over existing models, and this is despite the fact that the model was fine-tuned using a small labeled dataset.

This research investigates the potential application of generative language models, especially ChatGPT, in aviation safety analysis as a means to enhance the efficiency of safety analyses and accelerate the time it takes to process incident reports. In particular, ChatGPT was leveraged to generate incident synopses from narratives, which were subsequently compared with ground-truth synopses from the Aviation Safety Reporting System (ASRS) dataset. The comparison was facilitated by using embeddings from Large Language Models (LLMs), with aeroBERT demonstrating the highest similarity due to its aerospace-specific fine-tuning. A positive correlation was observed between the synopsis length and its cosine similarity. In a subsequent phase, human factors issues involved in incidents, as identified by ChatGPT, were compared to human factors issues identified by safety analysts. The precision was found to be 0.61, with ChatGPT demonstrating a cautious approach toward attributing human factors issues. Finally, the model was utilized to execute an evaluation of accountability. As no dedicated ground-truth column existed for this task, a manual evaluation was conducted to compare the quality of outputs provided by ChatGPT to the ground truths provided by safety analysts. This study discusses the advantages and pitfalls of generative language models in the context of aviation safety analysis and proposes a human-in-the-loop system to ensure responsible and effective utilization of such models, leading to continuous improvement and fostering a collaborative approach in the aviation safety domain.

View Video Presentation: https://doi.org/10.2514/6.2023-2583.vid Organizations predominantly use natural language (NL) for requirements elicitation, development, and management since NL is easy to understand and use by stakeholders with varying levels of experience, unlike specialized model-based languages, which require training to use. However, NL requirements entail ambiguities associated with language, a tedious and error- prone manual examination process, difficulties verifying requirement completeness, and failure to recognize and use technical terms effectively. Modern complex systems warrant an integrated and holistic approach to their development to capture their numerous interrelationships. This need has spurred a trend toward model-centric approaches to engineering as compared to traditional document-based methods. Transitioning to Model-Based Systems Engineering (MBSE) requires standardized, machine-readable requirements, but the conversion of NL requirements into models is hindered by the ambiguities and inconsistencies inherent in NL. The field of Natural Language Processing (NLP) has shown promise in facilitating the conversion of NL requirements into a format that enables their standardization and use in a model-based environment. Named-Entities Recognition (NER), in particular, can be leveraged to help address the ambiguities in NL requirements. To be of value, however, a language model needs to be fine-tuned to recognize aerospace-specific terms. To that end, we developed an annotated aerospace corpus and fine-tuned the BERT language model on the corpus to create aeroBERT-NER: a new model for identifying named entities (NEs) in aerospace requirements. A comparison between aeroBERT-NER and BERTBASE-NER showed the improved performance of aeroBERT-NER in identifying NEs within a set of aerospace requirements. The NEs identified by the model were used to create a glossary to facilitate consistent use of technical terms and language in aerospace requirements.

The increased complexity of modern systems is calling for an integrated and comprehensive approach to system design and development and, in particular, a shift toward Model-Based Systems Engineering (MBSE) approaches for system design. The requirements that serve as the foundation for these intricate systems are still primarily expressed in Natural Language (NL), which can contain ambiguities and inconsistencies and suffer from a lack of structure that hinders their direct translation into models. The colossal developments in the field of Natural Language Processing (NLP), in general, and Large Language Models (LLMs), in particular, can serve as an enabler for the conversion of NL requirements into machine-readable requirements. Doing so is expected to facilitate their standardization and use in a model-based environment. This paper discusses a two-fold strategy for converting NL requirements into machine-readable requirements using language models. The first approach involves creating a requirements table by extracting information from free-form NL requirements. The second approach consists of an agile methodology that facilitates the identification of boilerplate templates for different types of requirements based on observed linguistic patterns. For this study, three different LLMs are utilized. Two of these models are fine-tuned versions of Bidirectional Encoder Representations from Transformers (BERTs), specifically, aeroBERT-NER and aeroBERT-Classifier, which are trained on annotated aerospace corpora. Another LLM, called flair/chunk-english, is utilized to identify sentence chunks present in NL requirements. All three language models are utilized together to achieve the standardization of requirements. The effectiveness of the methodologies is demonstrated through the semi-automated creation of boilerplates for requirements from Parts 23 and 25 of Title 14 Code of Federal Regulations (CFRs).

In this work, we establish that maintenance and inspection are a risk factor in helicopter accidents. Between 2005 and 2015, flawed maintenance and inspection were causal factors in 14% to 21% of helicopter accidents in the U.S. civil fleet. For these maintenance-related accidents, we examined the incubation time from when the maintenance error was committed to the time when it resulted in an accident. We found a significant clustering of maintenance accidents within a short number of flight-hours after maintenance was performed. Of these accidents, 31% of these accidents occurred within the first 10 flight-hours. This is reminiscent of infant mortality in reliability engineering, and we characterized it as maintenance error infant mortality. The last quartile of maintenance-related accidents occurred after 60 flight-hours following maintenance and inspection. We then examined the “physics of failures” underlying maintenance-related accidents and analyzed the prevalence of different types of maintenance errors in helicopter accidents. We found, for instance, that the improper or incomplete (re)assembly or installation of a part category accounted for the majority of maintenance errors with 57% of such cases, and within this category, the incorrect torquing of the B-nut and incomplete assembly of critical linkages were the most prevalent maintenance errors. We also found that within the failure to perform a required preventive maintenance and inspection task category, the majority of the maintenance programs were not executed in compliance with federal regulations, nor with the manufacturer maintenance plan. Maintenance-related accidents are particularly hurtful for the rotorcraft community, and they can be eliminated. This is a reachable objective when technical competence meets organizational proficiency and the collective will of all the stakeholders in this community. We conclude with a set of recommendations based on our findings, which borrow from the ideas underlying the defense-in-depth safety principle to address this disquieting problem.