For many biological systems, a variety of simulation models exist. A new simulation model is rarely developed from scratch, but rather revises and extends an existing one. A key challenge, however, is to decide which model might be an appropriate starting point for a particular problem and why. To answer this question, we need to identify entities and activities that contributed to the development of a simulation model. Therefore, we exploit the provenance data model, PROV-DM, of the World Wide Web Consortium and, building on previous work, continue developing a PROV ontology for simulation studies. Based on a case study of 19 Wnt/β-catenin signaling models, we identify crucial entities and activities as well as useful metadata to both capture the provenance information from individual simulation studies and relate these forming a family of models. The approach is implemented in WebProv, a web application for inserting and querying provenance information. Our specialization of PROV-DM contains the entities Research Question, Assumption, Requirement, Qualitative Model, Simulation Model, Simulation Experiment, Simulation Data, and Wet-lab Data as well as activities referring to building, calibrating, validating, and analyzing a simulation model. We show that most Wnt simulation models are connected to other Wnt models by using (parts of) these models. However, the overlap, especially regarding the Wet-lab Data used for calibration or validation of the models is small. Making these aspects of developing a model explicit and queryable is an important step for assessing and reusing simulation models more effectively. Exposing this information helps to integrate a new simulation model within a family of existing ones and may lead to the development of more robust and valid simulation models. We hope that our approach becomes part of a standardization effort and that modelers adopt the benefits of provenance when considering or creating simulation models. We revise a provenance ontology for simulation studies of cellular biochemical models. Provenance information is useful for understanding the creation of a simulation model because it not only contains information about the entities and activities that have led to a simulation model but also their relations, all of which can be visualized. It provides additional structure by explicitly recording research questions, assumptions, and requirements and relating them along with data, qualitative models, simulation models, and simulation experiments through a small set of predefined but extensible activities. We have applied our concept to a family of 19 Wnt signaling models and implemented a web-based tool (WebProv) to store the provenance information from these studies. The resulting provenance graph visualizes the story line of simulation studies and demonstrates the creation and calibration of simulation models, the successive attempts of validation and extension, and shows, beyond an individual simulation study, how the Wnt models are related. Thereby, the steps and sources that contributed to a simulation model are made explicit. Our approach complements other approaches aimed at facilitating the reuse and assessment of simulation products in systems biology such as model repositories as well as annotation and documentation guidelines.

With the increasing complexity of simulation studies, and thus increasing complexity of simulation experiments, there is a high demand for better support for them to be conducted. Recently, model-driven approaches have been explored for facilitating the specification, execution, and reproducibility of simulation experiments. However, a more general approach that is suited for a variety of modeling and simulation areas, experiment types, and tools, which also allows for further automation, is still missing. Therefore, we present a novel model-driven engineering (MDE) framework for simulation studies that extends the state-of-the-art of conducting simulation experiments in the following ways: (a) Providing a structured representation of the various ingredients of simulation experiments in the form of meta models and collecting them in a repository improves knowledge sharing across application domains and simulation approaches. (b) Specifying simulation experiments in the quasi-standardized form of the meta models (e.g., via a GUI) and, subsequently, performing the automatic generation of experiment specifications in a language of choice increases both the productivity and quality of complex simulation experiments. (c) Automatic code transformation between specification languages via the meta models enables the reusability of simulation experiments. (d) Integrating the framework using a command-line interface allows for further automation of subprocesses within a simulation study. We demonstrate the advantages and practicality of our approach using real simulation studies from three different fields of simulation (stochastic discrete-event simulation of a cell signaling pathway, virtual prototyping of a neurostimulator, and finite element analysis of electric fields) and various experiment types (global sensitivity analysis, time course analysis, and convergence testing). The proposed framework can be the starting point for further automation of simulation experiments and, therefore, can assist in conducting simulation studies in a more systematic and effective manner. For example, based on this MDE framework, approaches for automatically selecting and parametrizing experimentation methods, or for planning follow-up activities depending on the context of the simulation study, could be developed.

Simulation has become, in many application areas, a sine qua non . Most recently, COVID-19 has underlined the importance of simulation studies and limitations in current practices and methods. We identify four goals of methodological work for addressing these limitations. The first is to provide better support for capturing, representing, and evaluating the context of simulation studies, including research questions, assumptions, requirements, and activities contributing to a simulation study. In addition, the composition of simulation models and other simulation studies’ products must be supported beyond syntactical coherence, including aspects of semantics and purpose, enabling their effective reuse. A higher degree of automating simulation studies will contribute to more systematic, standardized simulation studies and their efficiency. Finally, it is essential to invest increased effort into effectively communicating results and the processes involved in simulation studies to enable their use in research and decision making. These goals are not pursued independently of each other, but they will benefit from and sometimes even rely on advances in other sub-fields. In this article, we explore the basis and interdependencies evident in current research and practice and delineate future research directions based on these considerations.

Simulation experiments are typically conducted repeatedly during the model development process, for example, to revalidate if a behavioral property still holds after several model changes. Approaches for automatically reusing and generating simulation experiments can support modelers in conducting simulation studies in a more systematic and effective manner. They rely on explicit experiment specifications and, so far, on user interaction for initiating the reuse. Thereby, they are constrained to support the reuse of simulation experiments in a specific setting. Our approach now goes one step further by automatically identifying and adapting the experiments to be reused for a variety of scenarios. To achieve this, we exploit provenance graphs of simulation studies, which provide valuable information about the previous modeling and experimenting activities, and contain meta-information about the different entities that were used or produced during the simulation study. We define provenance patterns and associate them with a semantics, which allows us to interpret the different activities and construct transformation rules for provenance graphs. Our approach is implemented in a Reuse and Adapt framework for Simulation Experiments (RASE), which can interface with various modeling and simulation tools. In the case studies, we demonstrate the utility of our framework for (1) the repeated sensitivity analysis of an agent-based model of migration routes and (2) the cross-validation of two models of a cell signaling pathway.

Formal languages are an integral part of modeling and simulation. They allow the distillation of knowledge into concise simulation models amenable to automatic execution, interpretation, and analysis. However, the arguably most humanly accessible means of expressing models is through natural language, which is not easily interpretable by computers. Here, we evaluate how a Large Language Model (LLM) might be used for formalizing natural language into simulation models. Existing studies only explored using very large LLMs, like the commercial GPT models, without fine-tuning model weights. To close this gap, we show how an open-weights, 7B-parameter Mistral model can be fine-tuned to translate natural language descriptions to reaction network models in a domain-specific language, offering a self-hostable, compute-efficient, and memory efficient alternative. To this end, we develop a synthetic data generator to serve as the basis for fine-tuning and evaluation. Our quantitative evaluation shows that our fine-tuned Mistral model can recover the ground truth simulation model in up to 84.5% of cases. In addition, our small-scale user study demonstrates the model’s practical potential for one-time generation as well as interactive modeling in various domains. While promising, in its current form, the fine-tuned small LLM cannot catch up with large LLMs. We conclude that higher-quality training data are required, and expect future small and open-source LLMs to offer new opportunities.

Social simulation studies are complex. They typically combine various data sources and hypotheses about the system’s mechanisms that are integrated by intertwined processes of model building, simulation experiment execution and analysis. Various documentation approaches exist to increase the transparency and traceability of complex social simulation studies. Provenance standards enable the formalization of information on sources and activities, which contribute to the generation of an entity, in a queryable and computationally accessible manner. Provenance patterns can be defined as constraints on the relationships between specific types of activities and entities of a simulation study. In this paper, we refine the provenance pattern-based approach to address specific challenges of social agent-based simulation studies. Specifically, we focus on the activities and entities involved in collecting and analysing primary data about human decisions, and the collection and quality assessment of secondary data. We illustrate the potential of this approach by applying it to central activities and results of an agent-based simulation project and by presenting its implementation in a web-based tool.