Cover Image

To Code, or Not To Code? Exploring Impact of Code in Pre-training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Aryabumi, Viraat, Su, Yixuan, Ma, Raymond, Morisot, Adrien, Zhang, Ivan, Locatelli, Acyr, Fadaee, Marzieh, Üstün, Ahmet, Hooker, Sara
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
by: Mora, David, et al.
Published: (2025)

Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts
by: Gritsch, Nikolas, et al.
Published: (2024)

Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
by: D'souza, Daniel, et al.
Published: (2025)

One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
by: Abagyan, Diana, et al.
Published: (2025)

Verification Limits Code LLM Training
by: Gureja, Srishti, et al.
Published: (2025)

The Multilingual Divide and Its Impact on Global AI Safety
by: Peppin, Aidan, et al.
Published: (2025)

A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
by: Shimabucoro, Luisa, et al.
Published: (2025)

Aya 23: Open Weight Releases to Further Multilingual Progress
by: Aryabumi, Viraat, et al.
Published: (2024)

Tiny Aya: Bridging Scale and Multilingual Depth
by: Salamanca, Alejandro R., et al.
Published: (2026)

Aya Vision: Advancing the Frontier of Multilingual Multimodality
by: Dash, Saurabh, et al.
Published: (2025)

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
by: Üstün, Ahmet, et al.
Published: (2024)

The Disparate Impacts of Speculative Decoding
by: Sandler, Jameson, et al.
Published: (2025)

LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives
by: Shimabucoro, Luísa, et al.
Published: (2024)

Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning
by: Aakanksha, et al.
Published: (2024)

Text-to-Code Generation with Modality-relative Pre-training
by: Christopoulou, Fenia, et al.
Published: (2024)

From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions
by: Rakotonirina, Nathanaël Carraz, et al.
Published: (2025)

Efficient Benchmarking Is Just Feature Selection and Multiple Regression
by: Bowyer, Sam, et al.
Published: (2026)

Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
by: Dang, John, et al.
Published: (2024)

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
by: Aakanksha, et al.
Published: (2024)

Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement
by: Yu, Simon, et al.
Published: (2024)

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
by: Dang, John, et al.
Published: (2024)

The Leaderboard Illusion
by: Singh, Shivalika, et al.
Published: (2025)

How Does Quantization Affect Multilingual LLMs?
by: Marchisio, Kelly, et al.
Published: (2024)

Structure-aware Fine-tuning for Code Pre-trained Models
by: Wu, Jiayi, et al.
Published: (2024)

Fine-Tuning Pre-Trained Code Models for AI-Generated Code Detection
by: Ispas, Jany-Gabriel, et al.
Published: (2026)

CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
by: Tao, Leitian, et al.
Published: (2024)

Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance
by: Kim, Dongjun, et al.
Published: (2024)

Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models
by: Feng, Chengzhe, et al.
Published: (2024)

Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text
by: De Leon, Frances A. Laureano, et al.
Published: (2024)

IFEvalCode: Controlled Code Generation
by: Yang, Jian, et al.
Published: (2025)

Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
by: Nakamura, Taishi, et al.
Published: (2024)

Making, not Taking, the Best of N
by: Khairi, Ammar, et al.
Published: (2025)

Code Llama: Open Foundation Models for Code
by: Rozière, Baptiste, et al.
Published: (2023)

On Leakage of Code Generation Evaluation Datasets
by: Matton, Alexandre, et al.
Published: (2024)

Coding Triangle: How Does Large Language Model Understand Code?
by: Zhang, Taolin, et al.
Published: (2025)

CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL
by: Wang, Sijie, et al.
Published: (2025)

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms
by: Shi, Zhengyan, et al.
Published: (2024)

Exploring LLM Multi-Agents for ICD Coding
by: Li, Rumeng, et al.
Published: (2024)

Unlocking Reasoning Capability on Machine Translation in Large Language Models
by: Rajaee, Sara, et al.
Published: (2026)

Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation
by: Moon, Jiwon, et al.
Published: (2025)