publications

2025

1. Bridging Expressivity and Scalability with Adaptive Unitary SSMs

   Arjun Karuvally, Franz Nowak, Anderson T. Keller, and 3 more authors

   2025
2. Exponential Dynamic Energy Network for High Capacity Sequence Memory

   Arjun Karuvally, Pichsinee Lertsaroj, Terrence J. Sejnowski, and 1 more author

   2025
3. Reservoir Computation with Networks of Differentiating Neuron Ring Oscillators

   Alexander Yeung, Peter DelMastro, Arjun Karuvally, and 3 more authors

   Analytics, 2025

   Abs

   Reservoir computing is an approach to machine learning that leverages the dynamics of a complex system alongside a simple, often linear, machine learning model for a designated task. While many efforts have previously focused their attention on integrating neurons, which produce an output in response to large, sustained inputs, we focus on using differentiating neurons, which produce an output in response to large changes in input. Here, we introduce a small-world graph built from rings of differentiating neurons as a Reservoir Computing substrate. We find the coupling strength and network topology that enable these small-world networks to function as an effective reservoir. The dynamics of differentiating neurons naturally give rise to oscillatory dynamics when arranged in rings, where we study their computational use in the Reservoir Computing setting. We demonstrate the efficacy of these networks in the MNIST digit recognition task, achieving comparable performance of 90.65% to existing Reservoir Computing approaches. Beyond accuracy, we conduct systematic analysis of our reservoir’s internal dynamics using three complementary complexity measures that quantify neuronal activity balance, input dependence, and effective dimensionality. Our analysis reveals that optimal performance emerges when the reservoir operates with intermediate levels of neural entropy and input sensitivity, consistent with the edge-of-chaos hypothesis, where the system balances stability and responsiveness. The findings suggest that differentiating neurons can be a potential alternative to integrating neurons and can provide a sustainable future alternative for power-hungry AI applications.

2024

1. Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks

   Arjun Karuvally, Terrence Sejnowski, and Hava T Siegelmann

   In Proceedings of the 41st International Conference on Machine Learning, 21–27 jul 2024

   Abs PDF

   Traveling waves are a fundamental phenomenon in the brain, playing a crucial role in short-term information storage. In this study, we leverage the concept of traveling wave dynamics within a neural lattice to formulate a theoretical model of neural working memory in Recurrent Neural Networks (RNNs), study its properties, and its real world implications in AI. The proposed model diverges from traditional approaches, which assume information storage in static, register-like locations updated by interference. Instead, the model stores data as waves that is updated by the wave’s boundary conditions. We rigorously examine the model’s capabilities in representing and learning state histories, which are vital for learning history-dependent dynamical systems. The findings reveal that the model reliably stores external information and enhances the learning process by addressing the diminishing gradient problem of RNNs. To understand the model’s real-world applicability, we explore two cases: linear boundary condition and non-linear, self-attention-driven boundary condition. The experiments reveal that the linear scenario is effectively learned by RNNs through backpropagation when modeling history-dependent dynamical systems. Conversely, the non-linear scenario parallels an attention-only transformer. Collectively, our findings suggest the broader relevance of traveling waves in AI and its potential in advancing neural network architectures.

2023

1. 

   General Sequential Episodic Memory Model

   Arjun Karuvally, Terrence J. Sejnowski, and Hava T. Siegelmann

   In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, 21–27 jul 2023

   Abs PDF

   The state-of-the-art memory model is the General Associative Memory Model, a generalization of the classical Hopfield network. Like its ancestor, the general associative memory has a well-defined state-dependant energy surface, and its memories correlate with its fixed points. This is unlike human memories, which are commonly sequential rather than separated fixed points. In this paper, we introduce a class of General Sequential Episodic Memory Models (GSEMM) that, in the adiabatic limit, exhibit a dynamic energy surface, leading to a series of meta-stable states capable of encoding memory sequences. A multiple-timescale architecture enables the dynamic nature of the energy surface with newly introduced asymmetric synapses and signal propagation delays. We demonstrate its dense capacity under polynomial activation functions. GSEMM combines separate memories, short and long sequential episodic memories, under a unified theoretical framework, demonstrating how energy-based memory modeling can provide richer, human-like episodes.

2. 

   Episodic Memory Theory of Recurrent Neural Networks: Insights into Long-Term Information Storage and Manipulation

   Arjun Karuvally, Peter DelMastro, and Hava T. Siegelmann

   In Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML) at the 40th International Conference on Machine Learning ICML 2023, 21–27 jul 2023

   Abs PDF

   Recurrent neural networks (RNNs) have emerged as powerful models capable of storing and manipulating external information over long periods in various domains. Yet, the mechanisms that underly this behavior remain a mystery due to the black-box nature of these models. This paper addresses this question by proposing an episodic memory theory of RNN dynamics, enabling a more comprehensive understanding of the RNN weights as memories and inter-memory interactions. This approach sheds light on the inner workings of RNNs and connects to existing research on memory representation and organization. The theory extends the current linearization approaches by providing alternative interpretations of the eigenspectrum and its connection to the long-term storage and manipulation of information. We discuss how the segregation, representation, and composition of the variable binding problem—a fundamental question in cognitive science and artificial intelligence—can be mechanistically interpreted within the theory. Using an elementary task - repeat copy, we demonstrate the validity of the theory in experimental settings. Our work represents a step towards opening the black box of RNNs, offering new insights into their functionality and bridging the gap between recurrent neural networks and memory models.

2022

1. Energy-based General Sequential Episodic Memory Networks at the Adiabatic Limit

   Arjun Karuvally, Terry J. Sejnowski, and Hava T. Siegelmann

   CoRR, 21–27 jul 2022

   arXiv

2019

1. Learning proposals for sequential importance samplers using reinforced variational inference

   Zafarali Ahmed, Arjun Karuvally, Doina Precup, and 1 more author

   In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019, 21–27 jul 2019

   PDF

2018

1. Cache Miss Rate Predictability via Neural Networks

   Arjun Karuvally, Rishikesh Jha, Saket Tiwari, and 1 more author

   In Workshop on ML for Systems at NeurIPS 2018, 21–27 jul 2018

   PDF