Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons

Haider, Paul; Ellenberger, Benjamin; Kriener, Laura; Jordan, Jakob; Senn, Walter; Petrovici, Mihai A. (2021). Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons. Advances in Neural Information Processing Systems (NIPS), 35. MIT Press

Text NeurIPS-2021-latent-equilibrium-arbitrarily-fast-computation-with-arbitrarily-slow-neurons-Paper.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (2MB) | Request a copy

The response time of physical computational elements is finite, and neurons are no exception. In hierarchical models of cortical networks each layer thus introduces a response lag. This inherent property of physical dynamical systems results in delayed processing of stimuli and causes a timing mismatch between network output and instructive signals, thus afflicting not only inference, but also learning. We introduce Latent Equilibrium, a new framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential. This principle enables quasi-instantaneous inference independent of network depth and avoids the need for phased plasticity or computationally expensive network relaxation phases. We jointly derive disentangled neuron and synapse dynamics from a prospective energy function that depends on a network's generalized position and momentum. The resulting model can be interpreted as a biologically plausible approximation of error backpropagation in deep cortical networks with continuous-time, leaky neuronal dynamics and continuously active, local plasticity. We demonstrate successful learning of standard benchmark datasets, achieving competitive performance using both fully-connected and convolutional architectures, and show how our principle can be applied to detailed models of cortical microcircuitry. Furthermore, we study the robustness of our model to spatio-temporal substrate imperfections to demonstrate its feasibility for physical realization, be it in vivo or in silico.

Interest & Impact

Downloads

4 since deposited on 20 Apr 2022
4 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item