Image de couverture de livre

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Shi, Zhenmei, Ming, Yifei, Nguyen, Xuan-Phi, Liang, Yingyu, Joty, Shafiq
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Demystifying Domain-adaptive Post-training for Financial LLMs
par: Ke, Zixuan, et autres
Publié: (2025)

ParaICL: Towards Parallel In-Context Learning
par: Li, Xingxuan, et autres
Publié: (2024)

Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts
par: Nguyen, Xuan-Phi, et autres
Publié: (2023)

FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
par: Ming, Yifei, et autres
Publié: (2024)

Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
par: Pandit, Shrey, et autres
Publié: (2025)

Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms
par: Pandit, Shrey, et autres
Publié: (2025)

MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
par: Ke, Zixuan, et autres
Publié: (2025)

J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization
par: Xu, Austin, et autres
Publié: (2025)

SFR-RAG: Towards Contextually Faithful LLMs
par: Nguyen, Xuan-Phi, et autres
Publié: (2024)

NAACL2025 Tutorial: Adaptation of Large Language Models
par: Ke, Zixuan, et autres
Publié: (2025)

Can Language Models Compose Skills In-Context?
par: Liu, Zidong, et autres
Publié: (2025)

Towards Infinite-Long Prefix in Transformer
par: Liang, Yingyu, et autres
Publié: (2024)

Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflows
par: Ming, Yifei, et autres
Publié: (2025)

Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
par: Xu, Zhuoyan, et autres
Publié: (2024)

HSR-Enhanced Sparse Attention Acceleration
par: Chen, Bo, et autres
Publié: (2024)

Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings
par: Xu, Austin, et autres
Publié: (2025)

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
par: Liang, Yingyu, et autres
Publié: (2024)

Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
par: Xu, Austin, et autres
Publié: (2025)

MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks
par: Ke, Zixuan, et autres
Publié: (2026)

SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents
par: Nguyen, Xuan-Phi, et autres
Publié: (2025)

Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
par: Cao, Yang, et autres
Publié: (2024)

Why Larger Language Models Do In-context Learning Differently?
par: Shi, Zhenmei, et autres
Publié: (2024)

StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs
par: Chen, Hailin, et autres
Publié: (2024)

A Tighter Complexity Analysis of SparseGPT
par: Li, Xiaoyu, et autres
Publié: (2024)

On Positional Bias of Faithfulness for Long-form Summarization
par: Wan, David, et autres
Publié: (2024)

A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems
par: Ke, Zixuan, et autres
Publié: (2025)

Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization
par: Li, Chuyuan, et autres
Publié: (2025)

On Context Utilization in Summarization with Large Language Models
par: Ravaut, Mathieu, et autres
Publié: (2023)

Discovering Hidden Gems in Model Repositories
par: Kahana, Jonathan, et autres
Publié: (2026)

Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
par: Chen, Hailin, et autres
Publié: (2023)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
par: Liang, Yingyu, et autres
Publié: (2024)

RoPE Attention Can Be Trained in Almost Linear Time
par: Cao, Yang, et autres
Publié: (2024)

The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
par: Chen, Yifang, et autres
Publié: (2024)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
par: Li, Xiaoyu, et autres
Publié: (2024)

Neural Algorithmic Reasoning for Hypergraphs with Looped Transformers
par: Huang, Zekai, et autres
Publié: (2025)

Circuit Complexity Bounds for Visual Autoregressive Model
par: Ke, Yekun, et autres
Publié: (2025)

Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
par: Long, Do Xuan, et autres
Publié: (2025)

Unsupervised Summarization Re-ranking
par: Ravaut, Mathieu, et autres
Publié: (2022)

LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
par: Long, Do Xuan, et autres
Publié: (2024)

Shifting Long-Context LLMs Research from Input to Output
par: Wu, Yuhao, et autres
Publié: (2025)