Spatio-Temporal Credit Assignment in Neuronal Population Learning

Friedrich, Johannes; Urbanczik, Robert; Senn, Walter (2011). Spatio-Temporal Credit Assignment in Neuronal Population Learning. PLoS computational biology, 7(6), e1002092. San Francisco, Calif.: Public Library of Science 10.1371/journal.pcbi.1002092

[img]
Preview
Text
journal.pcbi.1002092.pdf - Published Version
Available under License Creative Commons: Attribution (CC-BY).

Download (566kB) | Preview

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Item Type:

Journal Article (Original Article)

Division/Institute:

04 Faculty of Medicine > Pre-clinic Human Medicine > Institute of Physiology

UniBE Contributor:

Friedrich, Johannes, Urbanczik, Robert, Senn, Walter

Subjects:

600 Technology > 610 Medicine & health

ISSN:

1553-734X

Publisher:

Public Library of Science

Language:

English

Submitter:

Factscience Import

Date Deposited:

04 Oct 2013 14:21

Last Modified:

05 Dec 2022 14:06

Publisher DOI:

10.1371/journal.pcbi.1002092

Web of Science ID:

000292381900033

Additional Information:

This work was supported by the Swiss National Science Foundation (SNSF, Sinergia grant CRSIKO-122697) and a grant from the Swiss SystemsX.ch initiative.

BORIS DOI:

10.7892/boris.6958

URI:

https://boris.unibe.ch/id/eprint/6958 (FactScience: 212089)

Actions (login required)

Edit item Edit item
Provide Feedback