2023-11-30 韓国基礎科学研究院(IBS)
(a) Diagram illustrating the ion channel activity in post-synaptic neurons. AMPA receptors are involved in the activation of post-synaptic neurons, while NMDA receptors are blocked by magnesium ions (Mg²⁺) but induce synaptic plasticity through the influx of calcium ions (Ca²⁺) when the post-synaptic neuron is sufficiently activated. (b) Flow diagram representing the computational process within the Transformer AI model. Information is processed sequentially through stages such as feed-forward layers, layer normalization, and self-attention layers. The graph depicting the current-voltage relationship of the NMDA receptors is very similar to the nonlinearity of the feed-forward layer. The input-output graph, based on the concentration of magnesium (α), shows the changes in the nonlinearity of the NMDA receptors.
◆この知見を基に、AIモデルの学習が神経科学の原則で説明できる可能性が浮かび上がり、低コストで高性能なAIシステムの開発に寄与するとともに、脳の仕組みに関する洞察が得られることが期待されています。
<関連情報>
- https://www.ibs.re.kr/cop/bbs/BBSMSTR_000000000738/selectBoardArticle.do
- https://openreview.net/forum?id=0z_cXcu1N6o
トランスフォーマーは長期記憶のためにNMDA受容体の非線形性を必要とします Transformer needs NMDA receptor nonlinearity for long-term memory
Dong-Kyum Kim, Jea Kwon, Meeyoung Cha, C. Justin Lee
OpenRebview Published: 02 Feb 2023
Abstract
The NMDA receptor (NMDAR) in the hippocampus is essential for learning and memory. We find an interesting resemblance between deep models’ nonlinear activation function and the NMDAR’s nonlinear dynamics. In light of a recent study that compared the transformer architecture to the formation of hippocampal memory, this paper presents new findings that NMDAR-like nonlinearity may be essential for consolidating short-term working memory into long-term reference memory. We design a navigation task assessing these two memory functions and show that manipulating the activation function (i.e., mimicking the Mg2+-gating of NMDAR) disrupts long-term memory formation. Our experimental data suggest that the concept of place cells and reference memory may reside in the feed-forward network layer of transformers and that nonlinearity plays a key role in these processes. Our findings propose that the transformer architecture and hippocampal spatial representation resemble by sharing the overlapping concept of NMDAR-like nonlinearity.
