20 January 2020 in IEEE Transactions on Neural Networks and Learning Systems
Temporal models based on recurrent neural networks have proven to be quite powerful in a wide variety of applications, including language modeling and speech processing. However, training these models often relies on backpropagation through time (BPTT), which entails unfolding the network over many time steps, making the process of conducting credit assignment considerably more challenging. Furthermore, the nature of backpropagation itself does not permit the use of nondifferentiable activation functions and is inherently sequential, making parallelization of the underlying training process difficult. Here, we propose the parallel temporal neural coding network (P-TNCN), a biologically inspired model trained by the learning algorithm we call local representation alignment. It aims to resolve the difficulties and problems that plague recurrent networks trained by BPTT. The architecture requires neither unrolling in time nor the derivatives of its internal activation functions. We compare our model and learning procedure with other BPTT alternatives (which also tend to be computationally expensive), including real-time recurrent learning, echo state networks, and unbiased online recurrent optimization. We show that it outperforms these on-sequence modeling benchmarks such as Bouncing MNIST, a new benchmark we denote as Bouncing NotMNIST, and Penn Treebank. Notably, our approach can, in some instances, outperform full BPTT as well as variants such as sparse attentive backtracking. Significantly, the hidden unit correction phase of P-TNCN allows it to adapt to new data sets even if its synaptic weights are held fixed (zero-shot adaptation) and facilitates retention of prior generative knowledge when faced with a task sequence. We present results that show the P-TNCN's ability to conduct zero-shot adaptation and online continual sequence modeling.
21 June 2023 in Nature Communications
Predicting failure in solids has broad applications including earthquake prediction which remains an unattainable goal. However, recent machine learning work shows that laboratory earthquakes can be predicted using micro-failure events and temporal evolution of fault zone elastic properties. Remarkably, these results come from purely data-driven models trained with large datasets. Such data are equivalent to centuries of fault motion rendering application to tectonic faulting unclear. In addition, the underlying physics of such predictions is poorly understood. Here, we address scalability using a novel Physics-Informed Neural Network (PINN). Our model encodes fault physics in the deep learning loss function using time-lapse ultrasonic data. PINN models outperform data-driven models and significantly improve transfer learning for small training datasets and conditions outside those used in training. Our work suggests that PINN offers a promising path for machine learning-based failure prediction and, ultimately for improving our understanding of earthquake physics and prediction.
1 December 2020 in IEEE Transactions on Artificial Intelligence
To learn complex formal grammars, recurrent neural networks (RNNs) require sufficient computational resources to ensure correct grammar recognition. One approach to expand model capacity is to couple an RNN to an external stack memory. Here, we introduce a “neural state” pushdown automaton (NSPDA), which consists of a discrete stack instead of an continuous one and is coupled to a neural network state machine. We empirically show its effectiveness in recognizing various context-free grammars (CFGs). First, we develop the underlying mechanics of the proposed higher order recurrent network and its manipulation of a stack as well as how to stably program its underlying pushdown automaton (PDA). We also introduce a noise regularization scheme for higher-order (tensor) networks and design an algorithm for improved incremental learning. Finally, we design a method for inserting grammar rules into a NSPDA and empirically show that this prior knowledge improves its training convergence time by an order of magnitude and, in some cases, leads to better generalization. The NSPDA is also compared to a classical analog stack neural network pushdown automaton (NNPDA) as well as a wide array of first and second-order RNNs with and without external memory, trained using different learning algorithms. Our results show that for the Dyck languages, prior rule-based knowledge is critical for optimization convergence and for ensuring generalization to longer sequences at test time. We observe that many RNNs with and without memory, but no prior knowledge, struggle to converge and generalize on complex and longer CFGs.
22 December 2023 in Information Sciences
6 March 2024 in Nature Communications
We link changes in crustal permeability to informative features of microearthquakes (MEQs) using two field hydraulic stimulation experiments where both MEQs and permeability evolution are recorded simultaneously. The Bidirectional Long Short-Term Memory (Bi-LSTM) model effectively predicts permeability evolution and ultimate permeability increase. Our findings confirm the form of key features linking the MEQs to permeability, offering mechanistically consistent interpretations of this association. Transfer learning correctly predicts permeability evolution of one experiment from a model trained on an alternate dataset and locale, which further reinforces the innate interdependency of permeability-to-seismicity. Models representing permeability evolution on reactivated fractures in both shear and tension suggest scaling relationships in which changes in permeability ( $$\Delta k$$ ) are linearly related to the seismic moment ( $$M$$ ) of individual MEQs as $$\Delta k\propto M$$ . This scaling relation rationalizes our observation of the permeability-to-seismicity linkage, contributes to its predictive robustness and accentuates its potential in characterizing crustal permeability evolution using MEQs.
16 September 2026 in PNAS Nexus
Predicting enzyme kinetics directly from sequence remains a central challenge in computational biology, particularly for mutations at catalytically essential residues. Existing models often default to wild-type predictions even when mutations cause substantial activity loss. This limits their utility for enzyme design and mechanistic inference. Here, we introduce CatRange, a machine learning framework trained on CatLog-27k, a human-in-the-loop, AI-agent curated dataset of 27,176 in vitro enzyme–substrate kinetic records. CatLog-27k was created through systematic audit and correction of BRENDA and SABIO-RK, with mutant entries manually reconciled against 2,158 source articles. CatRange reframes kinetic prediction as classification into log10-spaced bins for catalytic turnover (kcat) and substrate affinity (KM), rather than exact-value regression. This formulation reflects the order-of-magnitude scale at which enzyme kinetics are commonly interpreted. It reduces sensitivity to assay variability while preserving distinctions among functional catalytic and binding states. Using joint enzyme–substrate representations and gradient-boosted classifiers, CatRange predicts kinetic ranges for wild-type and mutant enzymes across held-out, out-of-distribution, and few-shot mutation settings. The model achieves robust order-of-magnitude recovery and class-balanced discrimination. Moreover, it captures mutation-induced transitions across kinetic regimes, including activity losses caused by perturbations of annotated catalytic residues. CatRange also identifies nonenzyme sequence inputs. Together, CatLog and CatRange provide an interpretable, mutation-sensitive framework for enzyme engineering and kinetic metabolic modeling.