Copertina

SkyLadder: Better and Faster Pretraining via Context Window Scheduling

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhu, Tongyao, Liu, Qian, Wang, Haonan, Chen, Shiqi, Gu, Xiangming, Pang, Tianyu, Kan, Min-Yen
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
di: Wang, Haonan, et al.
Pubblicazione: (2024)

Beyond Memorization: The Challenge of Random Memory Access in Language Models
di: Zhu, Tongyao, et al.
Pubblicazione: (2024)

CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization
di: Ding, Yixi, et al.
Pubblicazione: (2024)

Faster and Better LLMs via Latency-Aware Test-Time Scaling
di: Wang, Zili, et al.
Pubblicazione: (2025)

Better with Experience: Self-Evolving LLM Agents for Evidence-Grounded Health Community Notes
di: Fu, Zihang, et al.
Pubblicazione: (2026)

When Attention Sink Emerges in Language Models: An Empirical View
di: Gu, Xiangming, et al.
Pubblicazione: (2024)

Long-Context Language Modeling with Parallel Context Encoding
di: Yen, Howard, et al.
Pubblicazione: (2024)

Learning Word Embedding with Better Distance Weighting and Window Size Scheduling
di: Yang, Chaohao, et al.
Pubblicazione: (2024)

Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast
di: Gu, Xiangming, et al.
Pubblicazione: (2024)

ISQA: Informative Factuality Feedback for Scientific Summarization
di: Li, Zekai, et al.
Pubblicazione: (2024)

Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
di: Chen, Shiqi, et al.
Pubblicazione: (2025)

Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning
di: Wang, Haonan, et al.
Pubblicazione: (2025)

Better & Faster Large Language Models via Multi-token Prediction
di: Gloeckle, Fabian, et al.
Pubblicazione: (2024)

How to Train Long-Context Language Models (Effectively)
di: Gao, Tianyu, et al.
Pubblicazione: (2024)

From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
di: Wang, Haonan, et al.
Pubblicazione: (2025)

LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification
di: Yang, Penghui, et al.
Pubblicazione: (2025)

Better, Faster: Harnessing Self-Improvement in Large Reasoning Models
di: Zhong, Qihuang, et al.
Pubblicazione: (2026)

Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
di: Long, Do Xuan, et al.
Pubblicazione: (2025)

Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better
di: Zhao, Ji, et al.
Pubblicazione: (2026)

Prune as You Generate: Online Rollout Pruning for Faster and Better RLVR
di: Xu, Haobo, et al.
Pubblicazione: (2026)

Discursive Circuits: How Do Language Models Understand Discourse Relations?
di: Miao, Yisong, et al.
Pubblicazione: (2025)

Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
di: Huang, Chensen, et al.
Pubblicazione: (2024)

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
di: Huang, Chengsong, et al.
Pubblicazione: (2023)

Glyph: Scaling Context Windows via Visual-Text Compression
di: Cheng, Jiale, et al.
Pubblicazione: (2025)

A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
di: Tan, Naaman, et al.
Pubblicazione: (2024)

Extending LLMs' Context Window with 100 Samples
di: Zhang, Yikai, et al.
Pubblicazione: (2024)

Internalizing World Models via Self-Play Finetuning for Agentic RL
di: Chen, Shiqi, et al.
Pubblicazione: (2025)

Extending Context Window of Large Language Models from a Distributional Perspective
di: Wu, Yingsheng, et al.
Pubblicazione: (2024)

Fostering Video Reasoning via Next-Event Prediction
di: Wang, Haonan, et al.
Pubblicazione: (2025)

Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and Faster
di: Chen, Xiao, et al.
Pubblicazione: (2025)

PSC: Extending Context Window of Large Language Models via Phase Shift Calibration
di: Zhu, Wenqiao, et al.
Pubblicazione: (2025)

Establishing Task Scaling Laws via Compute-Efficient Model Ladders
di: Bhagia, Akshita, et al.
Pubblicazione: (2024)

Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
di: Warner, Benjamin, et al.
Pubblicazione: (2024)

FABSVer: Faster Training and Better Self-Verification for LLM Mathematical Reasoning
di: Pan, Haihui, et al.
Pubblicazione: (2026)

Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
di: Liu, Zijun, et al.
Pubblicazione: (2025)

Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
di: Yang, Zhaorui, et al.
Pubblicazione: (2024)

Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
di: Zhang, Xuan, et al.
Pubblicazione: (2024)

Improving Your Model Ranking on Chatbot Arena by Vote Rigging
di: Min, Rui, et al.
Pubblicazione: (2025)

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification
di: Liu, Shang, et al.
Pubblicazione: (2024)

Dynamic Masking Rate Schedules for MLM Pretraining
di: Ankner, Zachary, et al.
Pubblicazione: (2023)