Copertina

On Linear Representations and Pretraining Data Frequency in Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Merullo, Jack, Smith, Noah A., Wiegreffe, Sarah, Elazar, Yanai
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Evaluating $n$-Gram Novelty of Language Models Using Rusty-DAWG
di: Merrill, William, et al.
Pubblicazione: (2024)

Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
di: Wang, Xinyi, et al.
Pubblicazione: (2024)

Talking Heads: Understanding Inter-layer Communication in Transformer Language Models
di: Merullo, Jack, et al.
Pubblicazione: (2024)

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal
di: Cheng, Stephen, et al.
Pubblicazione: (2026)

Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior
di: Nadkarni, Rahul, et al.
Pubblicazione: (2025)

Mechanistic?
di: Saphra, Naomi, et al.
Pubblicazione: (2024)

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
di: Elazar, Yanai, et al.
Pubblicazione: (2023)

The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
di: Hase, Peter, et al.
Pubblicazione: (2024)

The Art of Saying No: Contextual Noncompliance in Language Models
di: Brahman, Faeze, et al.
Pubblicazione: (2024)

Gender Encoding Patterns in Pretrained Language Model Representations
di: Zakizadeh, Mahdi, et al.
Pubblicazione: (2025)

Paloma: A Benchmark for Evaluating Language Model Fit
di: Magnusson, Ian, et al.
Pubblicazione: (2023)

Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions
di: Wiegreffe, Sarah, et al.
Pubblicazione: (2024)

Everything is Plausible: Investigating the Impact of LLM Rationales on Human Notions of Plausibility
di: Palta, Shramay, et al.
Pubblicazione: (2025)

In-context Pretraining: Language Modeling Beyond Document Boundaries
di: Shi, Weijia, et al.
Pubblicazione: (2023)

Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
di: Palta, Shramay, et al.
Pubblicazione: (2024)

Tracing the Representation Geometry of Language Models from Pretraining to Post-training
di: Li, Melody Zixuan, et al.
Pubblicazione: (2025)

Estimating the Causal Effect of Early ArXiving on Paper Acceptance
di: Elazar, Yanai, et al.
Pubblicazione: (2023)

The Tokenization Bottleneck: How Vocabulary Extension Improves Chemistry Representation Learning in Pretrained Language Models
di: Kalamkar, Prathamesh, et al.
Pubblicazione: (2025)

LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv
di: Elazar, Yanai, et al.
Pubblicazione: (2026)

Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
di: Guo, Ping, et al.
Pubblicazione: (2025)

Foundation CAN LM: A Pretrained Language Model For Automotive CAN Data
di: Esashi, Akiharu, et al.
Pubblicazione: (2026)

Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning
di: Zhao, Yang, et al.
Pubblicazione: (2024)

Multilingual Pretraining for Pixel Language Models
di: Kesen, Ilker, et al.
Pubblicazione: (2025)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
di: Foroutan, Negar, et al.
Pubblicazione: (2025)

MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
di: Li, Jiazheng, et al.
Pubblicazione: (2025)

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
di: Boppana, Siddharth, et al.
Pubblicazione: (2026)

Representational Curvature Modulates Behavioral Uncertainty in Large Language Models
di: King, Jack, et al.
Pubblicazione: (2026)

Transferring Linear Features Across Language Models With Model Stitching
di: Chen, Alan, et al.
Pubblicazione: (2025)

BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
di: Manoj, Guduru, et al.
Pubblicazione: (2025)

Drop Dropout on Single-Epoch Language Model Pretraining
di: Liu, Houjun, et al.
Pubblicazione: (2025)

Rethinking the Role of Text Complexity in Language Model Pretraining
di: Velasco, Dan John, et al.
Pubblicazione: (2025)

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models
di: Lewis, Martha, et al.
Pubblicazione: (2022)

The Linear Representation Hypothesis and the Geometry of Large Language Models
di: Park, Kiho, et al.
Pubblicazione: (2023)

Representations as Language: An Information-Theoretic Framework for Interpretability
di: Conklin, Henry, et al.
Pubblicazione: (2024)

Evidence of Phase Transitions in Small Transformer-Based Language Models
di: Hong, Noah, et al.
Pubblicazione: (2025)

Emergent Introspective Awareness in Large Language Models
di: Lindsey, Jack
Pubblicazione: (2026)

PonderLM: Pretraining Language Models to Ponder in Continuous Space
di: Zeng, Boyi, et al.
Pubblicazione: (2025)

Code Pretraining Improves Entity Tracking Abilities of Language Models
di: Kim, Najoung, et al.
Pubblicazione: (2024)

Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
di: Elhady, Ahmed, et al.
Pubblicazione: (2025)

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
di: Yu, Huimu, et al.
Pubblicazione: (2024)