Cover Image

Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Haller, Patrick, Golde, Jonas, Akbik, Alan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptation
by: Haller, Patrick, et al.
Published: (2025)

BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models
by: Haller, Patrick, et al.
Published: (2024)

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
by: Aynetdinov, Ansar, et al.
Published: (2026)

Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs
by: Golde, Jonas, et al.
Published: (2023)

What Matters When Building Universal Multilingual Named Entity Recognition Models?
by: Golde, Jonas, et al.
Published: (2026)

PECC: Problem Extraction and Coding Challenges
by: Haller, Patrick, et al.
Published: (2024)

FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition
by: Golde, Jonas, et al.
Published: (2025)

PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models
by: Haller, Patrick, et al.
Published: (2025)

MastermindEval: A Simple But Scalable Reasoning Benchmark
by: Golde, Jonas, et al.
Published: (2025)

Pre-Training Curriculum for Multi-Token Prediction in Language Models
by: Aynetdinov, Ansar, et al.
Published: (2025)

Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data
by: Golde, Jonas, et al.
Published: (2024)

Large-Scale Label Interpretation Learning for Few-Shot Named Entity Recognition
by: Golde, Jonas, et al.
Published: (2024)

From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts
by: Christoph, Daniel, et al.
Published: (2025)

Question Decomposition for Retrieval-Augmented Generation
by: Ammann, Paul J. L., et al.
Published: (2025)

NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition
by: Merdjanovska, Elena, et al.
Published: (2024)

Efficient Streaming Language Models with Attention Sinks
by: Xiao, Guangxuan, et al.
Published: (2023)

When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models
by: You, Haoran, et al.
Published: (2024)

Efficiently Dispatching Flash Attention For Partially Filled Attention Masks
by: Sharma, Agniv, et al.
Published: (2024)

Efficient Attention Mechanisms for Large Language Models: A Survey
by: Sun, Yutao, et al.
Published: (2025)

Parallax: Parameterized Local Linear Attention for Language Modeling
by: Zuo, Yifei, et al.
Published: (2026)

FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models
by: Li, Chuan, et al.
Published: (2025)

LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling
by: Liu, Zeyu, et al.
Published: (2025)

PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
by: Chen, Lizhe, et al.
Published: (2025)

Bridging Textual and Tabular Worlds for Fact Verification: A Lightweight, Attention-Based Model
by: Varnosfaderani, Shirin Dabbaghi, et al.
Published: (2024)

ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models
by: Chen, Hao, et al.
Published: (2025)

Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
by: Zhang, Yang, et al.
Published: (2025)

Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
by: Pan, Dayan, et al.
Published: (2025)

Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
by: Hu, Xiang, et al.
Published: (2024)

Exploring Lightweight Large Language Models for Court View Generation
by: Hou, Zhitian, et al.
Published: (2026)

Lightweight Large Language Model for Medication Enquiry: Med-Pal
by: Elangovan, Kabilan, et al.
Published: (2024)

Sycophancy Hides Linearly in the Attention Heads
by: Genadi, Rifo, et al.
Published: (2026)

Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement
by: Zhou, Xiaofeng, et al.
Published: (2025)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
by: MiniCPM Team, et al.
Published: (2026)

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
by: Yang, Ning, et al.
Published: (2026)

Theory of Mind in Large Language Models: Assessment and Enhancement
by: Chen, Ruirui, et al.
Published: (2025)

Progressively Label Enhancement for Large Language Model Alignment
by: Liu, Biao, et al.
Published: (2024)

QUILL: Quotation Generation Enhancement of Large Language Models
by: Xiao, Jin, et al.
Published: (2024)

Attention Sinks in Diffusion Language Models
by: Rulli, Maximo Eduardo, et al.
Published: (2025)

TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks
by: Garbas, Lukas, et al.
Published: (2024)

S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models
by: He, Tao, et al.
Published: (2025)