RMAAT Accepted at ICLR 2026: Bringing Astrocyte Biology to Efficient Transformers

I am thrilled to announce that our paper, “RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers”, has been accepted at ICLR 2026!

The Problem: Quadratic Complexity

The quadratic complexity of the self-attention mechanism presents a significant impediment to applying Transformer models to long sequences. As sequence length grows, the computational cost and memory requirement of standard attention explode, limiting the applicability of Transformers in domains that require processing thousands of tokens efficiently.

Our Inspiration: Astrocytes

While most AI research focuses on neurons, the brain’s glial cells—especially astrocytes—play a critical, often overlooked role. Astrocytes modulate synaptic activity, consolidate memory, and regulate information flow. Our work explores how these biological principles can inform more efficient AI architectures.

RMAAT: The Architecture

RMAAT (Recurrent Memory Augmented Astromorphic Transformer) introduces several key innovations:

1. Segment-Based Processing with Persistent Memory

Instead of attending to the entire sequence at once, RMAAT divides the input into segments. A set of persistent memory tokens is maintained across segments, propagating essential contextual information in a recurrent manner. This transforms the single-pass attention into an iterative, memory-augmented process.

2. Astrocyte-Inspired Retention Factor (Long-Term Plasticity)

How does the model decide what information to store in its memory tokens? We draw inspiration from astrocyte Long-Term Plasticity (LTP). A novel retention factor is derived from simulated astrocyte dynamics. This factor adaptively compresses the memory representation, retaining critical information while discarding redundancy.

3. Linear-Complexity Attention (Short-Term Plasticity)

Within each segment, we replace the standard O(N^2) attention with an efficient linear-complexity mechanism. This is inspired by astrocyte Short-Term Plasticity (STP), which modulates rapid synaptic activity. The result is a significant speedup without sacrificing representational power.

4. Astrocytic Memory Replay Backpropagation (AMRB)

Training recurrent models is notoriously memory-intensive due to backpropagation through time. We introduce AMRB (Astrocytic Memory Replay Backpropagation), a novel algorithm designed specifically for memory efficiency in our recurrent framework.

Experimental Results

We evaluated RMAAT on the Long Range Arena (LRA) benchmark, a challenging suite of tasks designed to test long-context modeling.

Metric	RMAAT
Accuracy	Competitive with Transformers
Memory Usage	Substantially Reduced
Compute Time	Substantially Reduced

These results indicate the significant potential of incorporating astrocyte-inspired dynamics into scalable sequence models.

Conclusion

RMAAT demonstrates that looking beyond neurons to other components of biological intelligence can unlock new architectural insights for AI. We believe this work opens a promising direction for efficient, scalable, and biologically plausible sequence models.

Read the Full Paper: