Cover Image

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shi, Zhenmei, Ming, Yifei, Nguyen, Xuan-Phi, Liang, Yingyu, Joty, Shafiq
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Demystifying Domain-adaptive Post-training for Financial LLMs
by: Ke, Zixuan, et al.
Published: (2025)

ParaICL: Towards Parallel In-Context Learning
by: Li, Xingxuan, et al.
Published: (2024)

Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts
by: Nguyen, Xuan-Phi, et al.
Published: (2023)

FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
by: Ming, Yifei, et al.
Published: (2024)

Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
by: Pandit, Shrey, et al.
Published: (2025)

Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms
by: Pandit, Shrey, et al.
Published: (2025)

MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
by: Ke, Zixuan, et al.
Published: (2025)

J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
by: Xu, Austin, et al.
Published: (2025)

SFR-RAG: Towards Contextually Faithful LLMs
by: Nguyen, Xuan-Phi, et al.
Published: (2024)

NAACL2025 Tutorial: Adaptation of Large Language Models
by: Ke, Zixuan, et al.
Published: (2025)

Can Language Models Compose Skills In-Context?
by: Liu, Zidong, et al.
Published: (2025)

Towards Infinite-Long Prefix in Transformer
by: Liang, Yingyu, et al.
Published: (2024)

Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows
by: Ming, Yifei, et al.
Published: (2025)

Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
by: Xu, Zhuoyan, et al.
Published: (2024)

HSR-Enhanced Sparse Attention Acceleration
by: Chen, Bo, et al.
Published: (2024)

Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings
by: Xu, Austin, et al.
Published: (2025)

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
by: Liang, Yingyu, et al.
Published: (2024)

Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
by: Xu, Austin, et al.
Published: (2025)

MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks
by: Ke, Zixuan, et al.
Published: (2026)

SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
by: Nguyen, Xuan-Phi, et al.
Published: (2025)

Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

Why Larger Language Models Do In-context Learning Differently?
by: Shi, Zhenmei, et al.
Published: (2024)

StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs
by: Chen, Hailin, et al.
Published: (2024)

A Tighter Complexity Analysis of SparseGPT
by: Li, Xiaoyu, et al.
Published: (2024)

On Positional Bias of Faithfulness for Long-form Summarization
by: Wan, David, et al.
Published: (2024)

A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
by: Ke, Zixuan, et al.
Published: (2025)

Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization
by: Li, Chuyuan, et al.
Published: (2025)

On Context Utilization in Summarization with Large Language Models
by: Ravaut, Mathieu, et al.
Published: (2023)

Discovering Hidden Gems in Model Repositories
by: Kahana, Jonathan, et al.
Published: (2026)

Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
by: Chen, Hailin, et al.
Published: (2023)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
by: Liang, Yingyu, et al.
Published: (2024)

RoPE Attention Can Be Trained in Almost Linear Time
by: Cao, Yang, et al.
Published: (2024)

The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
by: Chen, Yifang, et al.
Published: (2024)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
by: Li, Xiaoyu, et al.
Published: (2024)

Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers
by: Huang, Zekai, et al.
Published: (2025)

Circuit Complexity Bounds for Visual Autoregressive Model
by: Ke, Yekun, et al.
Published: (2025)

Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
by: Long, Do Xuan, et al.
Published: (2025)

Unsupervised Summarization Re-ranking
by: Ravaut, Mathieu, et al.
Published: (2022)

LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
by: Long, Do Xuan, et al.
Published: (2024)

Shifting Long-Context LLMs Research from Input to Output
by: Wu, Yuhao, et al.
Published: (2025)