Cover Image

How Does Code Pretraining Affect Language Model Task Performance?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Petty, Jackson, van Steenkiste, Sjoerd, Linzen, Tal
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Impact of Depth on Compositional Generalization in Transformer Language Models
by: Petty, Jackson, et al.
Published: (2023)

A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
by: Eisape, Tiwalayo, et al.
Published: (2023)

Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
by: Hu, Michael Y., et al.
Published: (2025)

Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction
by: Petty, Jackson, et al.
Published: (2026)

Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models
by: Qiu, Linlu, et al.
Published: (2025)

In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
by: Mueller, Aaron, et al.
Published: (2023)

The Illusion of State in State-Space Models
by: Merrill, William, et al.
Published: (2024)

How Does Quantization Affect Multilingual LLMs?
by: Marchisio, Kelly, et al.
Published: (2024)

RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Context
by: Petty, Jackson, et al.
Published: (2025)

Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks
by: Timoneda, Joan C., et al.
Published: (2025)

Rapid Word Learning Through Meta In-Context Learning
by: Wang, Wentao, et al.
Published: (2025)

Do Language Models' Words Refer?
by: Mandelkern, Matthew, et al.
Published: (2023)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
by: Grangier, David, et al.
Published: (2024)

Language Models Improve When Pretraining Data Matches Target Tasks
by: Mizrahi, David, et al.
Published: (2025)

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
by: Hu, Michael Y., et al.
Published: (2026)

Zero-Shot Detection of LLM-Generated Code via Approximated Task Conditioning
by: Ashkenazi, Maor, et al.
Published: (2025)

How Does Response Length Affect Long-Form Factuality
by: Zhao, James Xu, et al.
Published: (2025)

Procedural Pretraining: Warming Up Language Models with Abstract Data
by: Jiang, Liangze, et al.
Published: (2026)

Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing
by: Jain, Manish, et al.
Published: (2025)

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes
by: Bhasin, Harmon, et al.
Published: (2024)

Textually Pretrained Speech Language Models
by: Hassid, Michael, et al.
Published: (2023)

Entailment Semantics Can Be Extracted from an Ideal Language Model
by: Merrill, William, et al.
Published: (2022)

How to Spin an Object: First, Get the Shape Right
by: Kabra, Rishabh, et al.
Published: (2024)

Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks
by: Fauber, Ben
Published: (2024)

Factual Consistency of Multilingual Pretrained Language Models
by: Fierro, Constanza, et al.
Published: (2022)

Does Differential Privacy Impact Bias in Pretrained NLP Models?
by: Islam, Md. Khairul, et al.
Published: (2024)

Scaling Laws for Downstream Task Performance of Large Language Models
by: Isik, Berivan, et al.
Published: (2024)

Unsupervised Pretraining for Fact Verification by Language Model Distillation
by: Bazaga, Adrián, et al.
Published: (2023)

From Lists to Emojis: How Format Bias Affects Model Alignment
by: Zhang, Xuanchang, et al.
Published: (2024)

Optimizing Multi-Task Learning for Enhanced Performance in Large Language Models
by: Qi, Zhen, et al.
Published: (2024)

SPAWNing Structural Priming Predictions from a Cognitively Motivated Parser
by: Prasad, Grusha, et al.
Published: (2024)

The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models
by: Bhaskar, Adithya, et al.
Published: (2024)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
by: Ruis, Laura, et al.
Published: (2024)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
by: Behnia, Tina, et al.
Published: (2025)

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
by: Bal, Melis Ilayda, et al.
Published: (2025)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
by: Sam, Dylan, et al.
Published: (2025)

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
by: Dong, Guanting, et al.
Published: (2023)

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
by: Yu, Zichun, et al.
Published: (2025)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
by: Chen, Zhuo, et al.
Published: (2026)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
by: Tanaka, Tatsunori, et al.
Published: (2025)