Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining

Documenti analoghi

• Causal Abstraction in Model Interpretability: A Compact Survey
  di: Zhang, Yihao
  Pubblicazione: (2024)
• Generating Pretraining Tokens from Organic Data for Data-Bound Scaling
  di: Yu, Zichun, et al.
  Pubblicazione: (2026)
• To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
  di: Singh, Karan, et al.
  Pubblicazione: (2026)
• Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
  di: Munkhdalai, Tsendsuren, et al.
  Pubblicazione: (2024)
• SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
  di: S, Santhosh G, et al.
  Pubblicazione: (2025)