Cover Image

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chitsaz, Kamran, Fournier, Quentin, Mordido, Gonçalo, Chandar, Sarath
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

NovoMolGen: Rethinking Molecular Language Model Pretraining
by: Chitsaz, Kamran, et al.
Published: (2025)

Lookbehind-SAM: k steps back, 1 step forward
by: Mordido, Gonçalo, et al.
Published: (2023)

Why Don't Prompt-Based Fairness Metrics Correlate?
by: Zayed, Abdelrahman, et al.
Published: (2024)

Should We Attend More or Less? Modulating Attention for Fairness
by: Zayed, Abdelrahman, et al.
Published: (2023)

CoPeP: Benchmarking Continual Pretraining for Protein Language Models
by: Patil, Darshan, et al.
Published: (2026)

Manifold Metric: A Loss Landscape Approach for Predicting Model Performance
by: Malviya, Pranshu, et al.
Published: (2024)

Torque-Aware Momentum
by: Malviya, Pranshu, et al.
Published: (2024)

The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning
by: Aghajohari, Milad, et al.
Published: (2025)

Promoting Exploration in Memory-Augmented Adam using Critical Momenta
by: Malviya, Pranshu, et al.
Published: (2023)

Faithfulness Measurable Masked Language Models
by: Madsen, Andreas, et al.
Published: (2023)

Are self-explanations from Large Language Models faithful?
by: Madsen, Andreas, et al.
Published: (2024)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
by: Abbes, Istabrak, et al.
Published: (2025)

Context-Aware Assistant Selection for Improved Inference Acceleration with Large Language Models
by: Huang, Jerry, et al.
Published: (2024)

Small Encoders Can Rival Large Decoders in Detecting Groundedness
by: Abbes, Istabrak, et al.
Published: (2025)

Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models
by: Prato, Gabriele, et al.
Published: (2025)

Mastering Memory Tasks with World Models
by: Samsami, Mohammad Reza, et al.
Published: (2024)

Steering Large Language Model Activations in Sparse Spaces
by: Bayat, Reza, et al.
Published: (2025)

Sub-goal Distillation: A Method to Improve Small Language Agents
by: Hashemzadeh, Maryam, et al.
Published: (2024)

Do Large Language Models Know How Much They Know?
by: Prato, Gabriele, et al.
Published: (2025)

Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
by: Nilaksh, et al.
Published: (2026)

The Expressive Limits of Diagonal SSMs for State-Tracking
by: Shakerinava, Mehran, et al.
Published: (2026)

Intelligent Switching for Reset-Free RL
by: Patil, Darshan, et al.
Published: (2024)

Neural Coherence : Find higher performance to out-of-distribution tasks from few samples
by: Guiroy, Simon, et al.
Published: (2025)

BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning
by: Zholus, Artem, et al.
Published: (2024)

CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Design
by: Govindarajan, Prashant, et al.
Published: (2025)

Towards Practical Tool Usage for Continually Learning LLMs
by: Huang, Jerry, et al.
Published: (2024)

Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning
by: Nilaksh, et al.
Published: (2026)

Too Big to Fool: Resisting Deception in Language Models
by: Samsami, Mohammad Reza, et al.
Published: (2024)

Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing
by: Hashemzadeh, Maryam, et al.
Published: (2026)

Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
by: Pham, Cuong, et al.
Published: (2025)

Interpretability Needs a New Paradigm
by: Madsen, Andreas, et al.
Published: (2024)

GRPO-$λ$: Credit Assignment improves LLM Reasoning
by: Parthasarathi, Prasanna, et al.
Published: (2025)

Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning
by: Huang, Jerry, et al.
Published: (2026)

CrystalGym: A New Benchmark for Materials Discovery Using Reinforcement Learning
by: Govindarajan, Prashant, et al.
Published: (2025)

Development of Pre-Trained Transformer-based Models for the Nepali Language
by: Thapa, Prajwal, et al.
Published: (2024)

Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization
by: Agrawal, Amey, et al.
Published: (2023)

Partially Rewriting a Transformer in Natural Language
by: Paulo, Gonçalo, et al.
Published: (2025)

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
by: Chen, Mengzhao, et al.
Published: (2024)

Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgrids
by: Nekoei, Hadi, et al.
Published: (2025)

Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
by: Huang, Jerry, et al.
Published: (2024)