Cover Image

Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Martinez, Richard Diehl, Goriely, Zebulon, Caines, Andrew, Buttery, Paula, Beinborn, Lisa
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes
by: Goriely, Zébulon, et al.
Published: (2024)

Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies
by: Salhan, Suchir, et al.
Published: (2024)

IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling
by: Goriely, Zébulon, et al.
Published: (2025)

BabyLM's First Words: Word Segmentation as a Phonological Probing Task
by: Goriely, Zébulon, et al.
Published: (2025)

What is the Best Sequence Length for BABYLM?
by: Salhan, Suchir, et al.
Published: (2025)

Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasing
by: Trhlik, Filip, et al.
Published: (2026)

ByteSpan: Information-Driven Subword Tokenisation
by: Goriely, Zébulon, et al.
Published: (2025)

Tending Towards Stability: Convergence Challenges in Small Language Models
by: Martinez, Richard Diehl, et al.
Published: (2024)

BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models
by: Gao, Yuan, et al.
Published: (2025)

The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement
by: Kamp, Jonathan, et al.
Published: (2024)

Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
by: Ganescu, Bianca-Mihaela, et al.
Published: (2025)

Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models
by: Weiss, Yuval, et al.
Published: (2025)

Learning Dynamics of Meta-Learning in Small Model Pretraining
by: Africa, David Demitri, et al.
Published: (2025)

Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
by: Africa, David Demitri, et al.
Published: (2025)

Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
by: Martinez, Richard Diehl, et al.
Published: (2025)

DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large Language Models
by: Thorat, Shantanu, et al.
Published: (2025)

Teacher Demonstrations in a BabyLM's Zone of Proximal Development for Contingent Multi-Turn Interaction
by: Salhan, Suchir, et al.
Published: (2025)

Learning from Sufficient Rationales: Analysing the Relationship Between Explanation Faithfulness and Token-level Regularisation Strategies
by: Kamp, Jonathan, et al.
Published: (2025)

Prompting open-source and commercial language models for grammatical error correction of English learner text
by: Davis, Christopher, et al.
Published: (2024)

Projective Methods for Mitigating Gender Bias in Pre-trained Language Models
by: Dawkins, Hillary, et al.
Published: (2024)

Once Upon a Time: Interactive Learning for Storytelling with Small Language Models
by: Martins, Jonas Mayer, et al.
Published: (2025)

What makes a word hard to learn? Modeling L1 influence on English vocabulary difficulty
by: Martins, Jonas Mayer, et al.
Published: (2026)

TriCon-Fair: Triplet Contrastive Learning for Mitigating Social Bias in Pre-trained Language Models
by: Lyu, Chong, et al.
Published: (2025)

Mitigating the Bias of Large Language Model Evaluation
by: Zhou, Hongli, et al.
Published: (2024)

Web(er) of Hate: A Survey on How Hate Speech Is Typed
by: Wang, Luna, et al.
Published: (2025)

Vocabulary shapes cross-lingual variation of word-order learnability in language models
by: Martins, Jonas Mayer, et al.
Published: (2026)

BiasDPO: Mitigating Bias in Language Models through Direct Preference Optimization
by: Allam, Ahmed
Published: (2024)

Open-DeBias: Toward Mitigating Open-Set Bias in Language Models
by: Rani, Arti, et al.
Published: (2025)

Mitigating Label Length Bias in Large Language Models
by: Sanz-Guerrero, Mario, et al.
Published: (2025)

Mitigating Biases in Language Models via Bias Unlearning
by: Liu, Dianqing, et al.
Published: (2025)

BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing
by: Roy, Subhro, et al.
Published: (2022)

Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models
by: Kryvosheieva, Daria, et al.
Published: (2024)

Understanding Syntactic Generalization in Structure-inducing Language Models
by: Arps, David, et al.
Published: (2025)

REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting
by: Huang, Nannan, et al.
Published: (2025)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
by: Guo, Xu, et al.
Published: (2026)

Do Multilingual Large Language Models Mitigate Stereotype Bias?
by: Nie, Shangrui, et al.
Published: (2024)

Detection, Classification, and Mitigation of Gender Bias in Large Language Models
by: Cheng, Xiaoqing, et al.
Published: (2025)

Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations
by: Lindemann, Matthias, et al.
Published: (2024)

Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment
by: Gallagher, Daniel, et al.
Published: (2026)

Universal Syntactic Structures: Modeling Syntax for Various Natural Languages
by: Kim, Min K., et al.
Published: (2023)