Cover Image

Analysing The Impact of Sequence Composition on Language Model Pre-Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhao, Yu, Qu, Yuanbin, Staniszewski, Konrad, Tworkowski, Szymon, Liu, Wei, Miłoś, Piotr, Wu, Yuxiang, Minervini, Pasquale
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Structured Packing in LLM Training Improves Long Context Utilization
by: Staniszewski, Konrad, et al.
Published: (2023)

Using Natural Language Explanations to Improve Robustness of In-context Learning
by: He, Xuanli, et al.
Published: (2023)

Analysing the Residual Stream of Language Models Under Knowledge Conflicts
by: Zhao, Yu, et al.
Published: (2024)

KV Cache Transform Coding for Compact Storage in LLM Inference
by: Staniszewski, Konrad, et al.
Published: (2025)

Inference-Time Hyper-Scaling with KV Cache Compression
by: Łańcucki, Adrian, et al.
Published: (2025)

Lightweight Latent Verifiers for Efficient Meta-Generation Strategies
by: Piotrowski, Bartosz, et al.
Published: (2025)

Self-Training Large Language Models for Tool-Use Without Demonstrations
by: Luo, Ne, et al.
Published: (2025)

Probing the Emergence of Cross-lingual Alignment during LLM Training
by: Wang, Hetong, et al.
Published: (2024)

Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish
by: Ruciński, Szymon
Published: (2024)

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression
by: Devoto, Alessio, et al.
Published: (2024)

FairBelief -- Assessing Harmful Beliefs in Language Models
by: Setzu, Mattia, et al.
Published: (2024)

Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models
by: Tyukin, Georgy, et al.
Published: (2024)

Answerability in Retrieval-Augmented Open-Domain Question Answering
by: Abdumalikov, Rustam, et al.
Published: (2024)

Noiser: Bounded Input Perturbations for Attributing Large Language Models
by: Madani, Mohammad Reza Ghasemi, et al.
Published: (2025)

SynDARin: Synthesising Datasets for Automated Reasoning in Low-Resource Languages
by: Ghazaryan, Gayane, et al.
Published: (2024)

SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations
by: Solano, Jesus, et al.
Published: (2023)

Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learning
by: Yu, Simon, et al.
Published: (2024)

PosterSum: A Multimodal Benchmark for Scientific Poster Summarization
by: Saxena, Rohit, et al.
Published: (2025)

Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints
by: Gema, Aryo Pradipta, et al.
Published: (2024)

Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain
by: Sayin, Burcu, et al.
Published: (2024)

Sequence-to-Sequence Spanish Pre-trained Language Models
by: Araujo, Vladimir, et al.
Published: (2023)

Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
by: Wu, Yu-Hang, et al.
Published: (2026)

AntLM: Bridging Causal and Masked Language Models
by: Yu, Xinru, et al.
Published: (2024)

Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs
by: Saxena, Rohit, et al.
Published: (2025)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
by: Guo, Xu, et al.
Published: (2026)

LLMLagBench: Identifying Temporal Training Boundaries in Large Language Models
by: Pęzik, Piotr, et al.
Published: (2025)

Evolution of Concepts in Language Model Pre-Training
by: Ge, Xuyang, et al.
Published: (2025)

The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
by: Hong, Giwon, et al.
Published: (2024)

Instruction Pre-Training: Language Models are Supervised Multitask Learners
by: Cheng, Daixuan, et al.
Published: (2024)

ConcEPT: Concept-Enhanced Pre-Training for Language Models
by: Wang, Xintao, et al.
Published: (2024)

Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication
by: Kuciński, Łukasz, et al.
Published: (2021)

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
by: Que, Haoran, et al.
Published: (2024)

Deciding not to Decide: Sound and Complete Effect Inference in the Presence of Higher-Rank Polymorphism
by: Balik, Patrycja, et al.
Published: (2025)

Remarks on Algebraic Reconstruction of Types and Effects
by: Balik, Patrycja, et al.
Published: (2026)

How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
by: Lu, Xin, et al.
Published: (2025)

Magnushammer: A Transformer-Based Approach to Premise Selection
by: Mikuła, Maciej, et al.
Published: (2023)

PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation
by: Kaing, Hour, et al.
Published: (2025)

SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models
by: Kim, SungHo, et al.
Published: (2026)

LoopRPT: Reinforcement Pre-Training for Looped Language Models
by: Tang, Guo, et al.
Published: (2026)

Edinburgh Clinical NLP at SemEval-2024 Task 2: Fine-tune your model unless you have access to GPT-4
by: Gema, Aryo Pradipta, et al.
Published: (2024)