AIの記憶形成メカニズムは脳のメカニズムと驚くほど似ていることが判明(AI’s memory-forming mechanism found to be strikingly similar to that of the brain)


2023-11-30 韓国基礎科学研究院(IBS)

図 1:(a)シナプス後ニューロンのイオンチャネル活動を示す図。 AMPA受容体はシナプス後ニューロンの活性化に関与しますが、NMDA受容体はマグネシウムイオン(Mg2⁺)によってブロックされますが、シナプス後ニューロンが十分に活性化されるとカルシウムイオン(Ca2⁺)の流入を通じてシナプス可塑性を誘導します。 (b) Transformer AI モデル内の計算プロセスを表すフロー図。 情報は、フィードフォワード層、層正規化、セルフアテンション層などの段階を経て順次処理されます。 NMDA 受容体の電流と電圧の関係を示すグラフは、フィードフォワード層の非線形性に非常に似ています。 マグネシウム濃度(α)に基づく入出力グラフは、NMDA受容体の非線形性の変化を示しています。 (a) Diagram illustrating the ion channel activity in post-synaptic neurons. AMPA receptors are involved in the activation of post-synaptic neurons, while NMDA receptors are blocked by magnesium ions (Mg²⁺) but induce synaptic plasticity through the influx of calcium ions (Ca²⁺) when the post-synaptic neuron is sufficiently activated. (b) Flow diagram representing the computational process within the Transformer AI model. Information is processed sequentially through stages such as feed-forward layers, layer normalization, and self-attention layers. The graph depicting the current-voltage relationship of the NMDA receptors is very similar to the nonlinearity of the feed-forward layer. The input-output graph, based on the concentration of magnesium (α), shows the changes in the nonlinearity of the NMDA receptors.



トランスフォーマーは長期記憶のためにNMDA受容体の非線形性を必要とします Transformer needs NMDA receptor nonlinearity for long-term memory

Dong-Kyum Kim, Jea Kwon, Meeyoung Cha, C. Justin Lee
OpenRebview  Published: 02 Feb 2023


The NMDA receptor (NMDAR) in the hippocampus is essential for learning and memory. We find an interesting resemblance between deep models’ nonlinear activation function and the NMDAR’s nonlinear dynamics. In light of a recent study that compared the transformer architecture to the formation of hippocampal memory, this paper presents new findings that NMDAR-like nonlinearity may be essential for consolidating short-term working memory into long-term reference memory. We design a navigation task assessing these two memory functions and show that manipulating the activation function (i.e., mimicking the Mg2+-gating of NMDAR) disrupts long-term memory formation. Our experimental data suggest that the concept of place cells and reference memory may reside in the feed-forward network layer of transformers and that nonlinearity plays a key role in these processes. Our findings propose that the transformer architecture and hippocampal spatial representation resemble by sharing the overlapping concept of NMDAR-like nonlinearity.