Imagen de Portada

In-context Pretraining: Language Modeling Beyond Document Boundaries

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Shi, Weijia, Min, Sewon, Lomeli, Maria, Zhou, Chunting, Li, Margaret, Szilvasy, Gergely, James, Rich, Lin, Xi Victoria, Smith, Noah A., Zettlemoyer, Luke, Yih, Scott, Lewis, Mike
Formato:	Preprint
Publicado:	2023
Materias:	Computation and Language Artificial Intelligence Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

RA-DIT: Retrieval-Augmented Dual Instruction Tuning
por: Lin, Xi Victoria, et al.
Publicado: (2023)

LMFusion: Adapting Pretrained Language Models for Multimodal Generation
por: Shi, Weijia, et al.
Publicado: (2024)

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
por: Min, Sewon, et al.
Publicado: (2023)

Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
por: Liang, Weixin, et al.
Publicado: (2024)

Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
por: Shao, Rulin, et al.
Publicado: (2024)

Self-Alignment with Instruction Backtranslation
por: Li, Xian, et al.
Publicado: (2023)

Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
por: Szilvasy, Gergely, et al.
Publicado: (2026)

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
por: Liu, Jiacheng, et al.
Publicado: (2024)

Detecting Pretraining Data from Large Language Models
por: Shi, Weijia, et al.
Publicado: (2023)

Instruction-tuned Language Models are Better Knowledge Learners
por: Jiang, Zhengbao, et al.
Publicado: (2024)

Do Membership Inference Attacks Work on Large Language Models?
por: Duan, Michael, et al.
Publicado: (2024)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
por: Ma, Xuezhe, et al.
Publicado: (2024)

Vector search with small radiuses
por: Szilvasy, Gergely, et al.
Publicado: (2024)

EMO: Pretraining Mixture of Experts for Emergent Modularity
por: Wang, Ryan, et al.
Publicado: (2026)

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
por: Hu, Yushi, et al.
Publicado: (2024)

Evaluating Copyright Takedown Methods for Language Models
por: Wei, Boyi, et al.
Publicado: (2024)

ReasonIR: Training Retrievers for Reasoning Tasks
por: Shao, Rulin, et al.
Publicado: (2025)

FlexOlmo: Open Language Models for Flexible Data Use
por: Shi, Weijia, et al.
Publicado: (2025)

Inference-time sparse attention with asymmetric indexing
por: Mazaré, Pierre-Emmanuel, et al.
Publicado: (2025)

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
por: Lin, Xi Victoria, et al.
Publicado: (2024)

Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
por: Singh, Aaditya K., et al.
Publicado: (2024)

(Mis)Fitting: A Survey of Scaling Laws
por: Li, Margaret, et al.
Publicado: (2025)

Beyond Language Modeling: An Exploration of Multimodal Pretraining
por: Tong, Shengbang, et al.
Publicado: (2026)

CAT: Content-Adaptive Image Tokenization
por: Shen, Junhong, et al.
Publicado: (2025)

ALMA: Alignment with Minimal Annotation
por: Yasunaga, Michihiro, et al.
Publicado: (2024)

Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
por: Gonen, Hila, et al.
Publicado: (2024)

Demystifying Prompts in Language Models via Perplexity Estimation
por: Gonen, Hila, et al.
Publicado: (2022)

Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
por: Blevins, Terra, et al.
Publicado: (2024)

Byte Latent Transformer: Patches Scale Better Than Tokens
por: Pagnoni, Artidoro, et al.
Publicado: (2024)

MUSE: Machine Unlearning Six-Way Evaluation for Language Models
por: Shi, Weijia, et al.
Publicado: (2024)

The Hidden Cost of Thinking: Energy Use and Environmental Impact of LMs Beyond Pretraining
por: Morrison, Jacob, et al.
Publicado: (2026)

The Faiss library
por: Douze, Matthijs, et al.
Publicado: (2024)

Short window attention enables long-term memorization
por: Cabannes, Loïc, et al.
Publicado: (2025)

Slicing and Dicing: Configuring Optimal Mixtures of Experts
por: Li, Margaret, et al.
Publicado: (2026)

Compute Optimal Tokenization
por: Limisiewicz, Tomasz, et al.
Publicado: (2026)

Stochastic activations
por: Lomeli, Maria, et al.
Publicado: (2025)

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation
por: Chen, Tong, et al.
Publicado: (2024)

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
por: He, Jacqueline, et al.
Publicado: (2026)

Memory Layers at Scale
por: Berges, Vincent-Pierre, et al.
Publicado: (2024)

Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
por: Morrison, Jacob, et al.
Publicado: (2026)