Deep learning is emerging in many industrial sectors in hand with big data analytics to streamline production. In the biomanufacturing sector, big data infrastructure is lagging compared to other industries. A promising approach is to combine deep neural networks (DNN) with prior knowledge in hybrid neural network (HNN) workflows that are less dependent on the quality and quantity of data. This paper reviews published articles over the past 30 years on the topic of HNN applications to bioprocesses. It reveals that HNNs have been applied to various bioprocesses, including microbial cultures, animal cells cultures, mixed microbial cultures, and enzyme biocatalysis. HNNs have been applied for process analysis, process monitoring, development of software sensors, open- and closed-loop control, batch-to-batch control, model predictive control, intensified design of experiments, quality-by-design, and recently for the development of digital twins. Most previous HNN studies have combined shallow feedforward neural networks (FFNNs) with physical laws, such as macroscopic material balance equations, following the semiparametric design principle. Only recently, deep HNNs based on deep FFNNs, convolution neural networks (CNN), long short-term memory (LSTM) networks and physics-informed neural networks (PINNs) have been reported. The biopharma sector is currently a major driver but applications to biologics quality attributes, new modalities, and downstream processing are significant research gaps.

Introduction: Hybrid modeling combining First-Principles with machine learning is becoming a pivotal methodology for Biopharma 4.0 enactment. Chinese Hamster Ovary (CHO) cells, being the workhorse for industrial glycoproteins production, have been the object of several hybrid modeling studies. Most previous studies pursued a shallow hybrid modeling approach based on three-layered Feedforward Neural Networks (FFNNs) combined with macroscopic material balance equations. Only recently, the hybrid modeling field is incorporating deep learning into its framework with significant gains in descriptive and predictive power.Methods: This study compares, for the first time, deep and shallow hybrid modeling in a CHO process development context. Data of 24 fed-batch cultivations of a CHO-K1 cell line expressing a target glycoprotein, comprising 30 measured state variables over time, were used to compare both methodologies. Hybrid models with varying FFNN depths (3-5 layers) were systematically compared using two training methodologies. The classical training is based on the Levenberg-Marquardt algorithm, indirect sensitivity equations and cross-validation. The deep learning is based on the Adaptive Moment Estimation Method (ADAM), stochastic regularization and semidirect sensitivity equations.Results and conclusion: The results point to a systematic generalization improvement of deep hybrid models over shallow hybrid models. Overall, the training and testing errors decreased by 14.0% and 23.6% respectively when applying the deep methodology. The Central Processing Unit (CPU) time for training the deep hybrid model increased by 31.6% mainly due to the higher FFNN complexity. The final deep hybrid model is shown to predict the dynamics of the 30 state variables within the error bounds in every test experiment. Notably, the deep hybrid model could predict the metabolic shifts in key metabolites (e.g., lactate, ammonium, glutamine and glutamate) in the test experiments. We expect deep hybrid modeling to accelerate the deployment of high-fidelity digital twins in the biopharma sector in the near future.

The combination of physical equations with deep learning is becoming a promising methodology for bioprocess digitalization. In this paper, we investigate for the first time the combination of long short‐term memory (LSTM) networks with first principles equations in a hybrid workflow to describe human embryonic kidney 293 (HEK293) culture dynamics. Experimental data of 27 extracellular state variables in 20 fed‐batch HEK293 cultures were collected in a parallel high throughput 250 mL cultivation system in an industrial process development setting. The adaptive moment estimation method with stochastic regularization and cross‐validation were employed for deep learning. A total of 784 hybrid models with varying deep neural network architectures, depths, layers sizes and node activation functions were compared. In most scenarios, hybrid LSTM models outperformed classical hybrid Feedforward Neural Network (FFNN) models in terms of training and testing error. Hybrid LSTM models revealed to be less sensitive to data resampling than FFNN hybrid models. As disadvantages, Hybrid LSTM models are in general more complex (higher number of parameters) and have a higher computation cost than FFNN hybrid models. The hybrid model with the highest prediction accuracy consisted in a LSTM network with seven internal states connected in series with dynamic material balance equations. This hybrid model correctly predicted the dynamics of the 27 state variables (R² = 0.93 in the test data set), including biomass, key substrates, amino acids and metabolic by‐products for around 10 cultivation days.

Flux balance analysis (FBA) is currently the standard method to compute metabolic fluxes in genome-scale networks. Several FBA extensions employing diverse objective functions and/or constraints have been published. Here we propose a hybrid semi-parametric FBA extension that combines mechanistic-level constraints (parametric) with empirical constraints (non-parametric) in the same linear program. A CHO dataset with 27 measured exchange fluxes obtained from 21 reactor experiments served to evaluate the method. The mechanistic constraints were deduced from a reduced CHO-K1 genome-scale network with 686 metabolites, 788 reactions and 210 degrees of freedom. The non-parametric constraints were obtained by principal component analysis of the flux dataset. The two types of constraints were integrated in the same linear program showing comparable computational cost to standard FBA. The hybrid FBA is shown to significantly improve the specific growth rate prediction under different constraints scenarios. A metabolically efficient cell growth feed targeting minimal byproducts accumulation was designed by hybrid FBA. It is concluded that integrating parametric and nonparametric constraints in the same linear program may be an efficient approach to reduce the solution space and to improve the predictive power of FBA methods when critical mechanistic information is missing.

In this paper, a computational framework is proposed that merges mechanistic modeling with deep neural networks obeying the Systems Biology Markup Language (SBML) standard. Over the last 20 years, the systems biology community has developed a large number of mechanistic models that are currently stored in public databases in SBML. With the proposed framework, existing SBML models may be redesigned into hybrid systems through the incorporation of deep neural networks into the model core, using a freely available python tool. The so-formed hybrid mechanistic/neural network models are trained with a deep learning algorithm based on the adaptive moment estimation method (ADAM), stochastic regularization and semidirect sensitivity equations. The trained hybrid models are encoded in SBML and uploaded in model databases, where they may be further analyzed as regular SBML models. This approach is illustrated with three well-known case studies: the Escherichia coli threonine synthesis model, the P58IPK signal transduction model, and the Yeast glycolytic oscillations model. The proposed framework is expected to greatly facilitate the widespread use of hybrid modeling techniques for systems biology applications.