Copertina

What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Awobade, Busayo, Oduwole, Mardiyyah, Kolawole, Steven
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

From Scarcity to Efficiency: Investigating the Effects of Data Augmentation on African Machine Translation
di: Oduwole, Mardiyyah, et al.
Pubblicazione: (2025)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
di: Magnusson, Ian, et al.
Pubblicazione: (2025)

Language Models Improve When Pretraining Data Matches Target Tasks
di: Mizrahi, David, et al.
Pubblicazione: (2025)

AfriVox-v2: A Domain-Verticalized Benchmark for In-the-Wild African Speech Recognition
di: Awobade, Busayo, et al.
Pubblicazione: (2026)

Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
di: Cao, Jinghan, et al.
Pubblicazione: (2026)

Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining
di: Huang, Ruizhe, et al.
Pubblicazione: (2025)

Is Training Data Quality or Quantity More Impactful to Small Language Model Performance?
di: Sajith, Aryan, et al.
Pubblicazione: (2024)

Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
di: Roger, Alexis, et al.
Pubblicazione: (2025)

When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
di: Galeone, Cosimo, et al.
Pubblicazione: (2026)

Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't
di: Dang, Quy-Anh, et al.
Pubblicazione: (2025)

From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When
di: Wibisono, Kevin Christian, et al.
Pubblicazione: (2024)

On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages
di: Shirke, Mayur, et al.
Pubblicazione: (2025)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
di: Behnia, Tina, et al.
Pubblicazione: (2025)

Small Language Models Improve Giants by Rewriting Their Outputs
di: Vernikos, Giorgos, et al.
Pubblicazione: (2023)

Self-Taught Self-Correction for Small Language Models
di: Moskvoretskii, Viktor, et al.
Pubblicazione: (2025)

An Analysis for Reasoning Bias of Language Models with Small Initialization
di: Yao, Junjie, et al.
Pubblicazione: (2025)

Smaller Language Models are Better Black-box Machine-Generated Text Detectors
di: Mireshghallah, Niloofar, et al.
Pubblicazione: (2023)

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
di: Sanyal, Sunny, et al.
Pubblicazione: (2024)

Procedural Pretraining: Warming Up Language Models with Abstract Data
di: Jiang, Liangze, et al.
Pubblicazione: (2026)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
di: Sam, Dylan, et al.
Pubblicazione: (2025)

Need a Small Specialized Language Model? Plan Early!
di: Grangier, David, et al.
Pubblicazione: (2024)

From Small to Large Language Models: Revisiting the Federalist Papers
di: Jeong, So Won, et al.
Pubblicazione: (2025)

Domain-Adaptive Continued Pre-Training of Small Language Models
di: Faroz, Salman
Pubblicazione: (2025)

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
di: Yang, Yu, et al.
Pubblicazione: (2024)

Stacking Small Language Models for Generalizability
di: Liang, Laurence
Pubblicazione: (2024)

All Language Models Large and Small
di: Chen, Zhixun, et al.
Pubblicazione: (2024)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
di: Chen, Zhuo, et al.
Pubblicazione: (2026)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
di: Tanaka, Tatsunori, et al.
Pubblicazione: (2025)

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
di: Ankner, Zachary, et al.
Pubblicazione: (2024)

Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?
di: Scaria, Nicy, et al.
Pubblicazione: (2024)

Domain-Adaptive Small Language Models for Structured Tax Code Prediction
di: Nath, Souvik, et al.
Pubblicazione: (2025)

Architectural Trade-offs in Small Language Models Under Compute Constraints
di: Bhatti, Shivraj Singh
Pubblicazione: (2025)

Small Models Are (Still) Effective Cross-Domain Argument Extractors
di: Gantt, William, et al.
Pubblicazione: (2024)

Training-free Ultra Small Model for Universal Sparse Reconstruction in Compressed Sensing
di: Tang, Chaoqing, et al.
Pubblicazione: (2025)

Small Language Models: Survey, Measurements, and Insights
di: Lu, Zhenyan, et al.
Pubblicazione: (2024)

Squat: Quant Small Language Models on the Edge
di: Shen, Xuan, et al.
Pubblicazione: (2024)

Towards Reasoning Ability of Small Language Models
di: Srivastava, Gaurav, et al.
Pubblicazione: (2025)

Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
di: Xie, Wanyun, et al.
Pubblicazione: (2025)

Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models
di: Zhao, Zheng, et al.
Pubblicazione: (2024)

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
di: Hu, Shengding, et al.
Pubblicazione: (2024)