Cover Image

TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Ruiyi, Somayajula, Sai Ashish, Xie, Pengtao
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoning
by: Cao, Qi, et al.
Published: (2025)

Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts
by: Somayajula, Sai Ashish, et al.
Published: (2024)

AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning
by: Zhang, Ruiyi, et al.
Published: (2024)

Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs
by: Gao, Xin, et al.
Published: (2025)

Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization
by: Guo, Han, et al.
Published: (2024)

DAJ: Data-Reweighted LLM Judge for Test-Time Scaling in Code Generation
by: Qin, Peijia, et al.
Published: (2026)

BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation
by: Qin, Peijia, et al.
Published: (2024)

Token-Specific Watermarking with Enhanced Detectability and Semantic Coherence for Large Language Models
by: Huo, Mingjia, et al.
Published: (2024)

BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Models
by: Qiang, Rushi, et al.
Published: (2024)

SteganoBackdoor: Stealthy and Data-Efficient Backdoor Attacks on Language Models
by: Xue, Eric, et al.
Published: (2025)

Send a SCOUT First: Pre-hoc Reasoning for Adaptive Detector Allocation in Prompt-Injection Defense
by: Zhang, Shuhao, et al.
Published: (2026)

Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining
by: Sow, Daouda, et al.
Published: (2025)

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights
by: Gan, Yulu, et al.
Published: (2026)

RLP: Reinforcement as a Pretraining Objective
by: Hatamizadeh, Ali, et al.
Published: (2025)

When Priors Backfire: On the Vulnerability of Unlearnable Examples to Pretraining
by: Li, Zhihao, et al.
Published: (2026)

DreamPRM-Code: Function-as-Step Process Reward Model with Label Correction for LLM Coding
by: Zhang, Ruiyi, et al.
Published: (2025)

DFedReweighting: A Unified Framework for Objective-Oriented Reweighting in Decentralized Federated Learning
by: Zhang, Kaichuang, et al.
Published: (2025)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
by: Grangier, David, et al.
Published: (2024)

Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models
by: Xue, Shuchen, et al.
Published: (2025)

Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning
by: Hu, Bokai, et al.
Published: (2024)

Practical Efficiency of Muon for Pretraining
by: AI, Essential, et al.
Published: (2025)

HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models
by: Zhou, Yu, et al.
Published: (2024)

FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generation
by: Zhang, Ruiyi, et al.
Published: (2026)

Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning
by: Cao, Qi, et al.
Published: (2026)

On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm
by: Zhou, Zhanpeng, et al.
Published: (2024)

Adaptive Pruning of Pretrained Transformer via Differential Inclusions
by: Ding, Yizhuo, et al.
Published: (2025)

Task-Agnostic Contrastive Pretraining for Relational Deep Learning
by: Peleška, Jakub, et al.
Published: (2025)

Locking Pretrained Weights via Deep Low-Rank Residual Distillation
by: Sakamoto, Keitaro, et al.
Published: (2026)

Pretrain-Test Task Alignment Governs Generalization in In-Context Learning
by: Letey, Mary I., et al.
Published: (2025)

Self-Supervised Pretraining for Aerial Road Extraction
by: Polley, Rupert, et al.
Published: (2025)

ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
by: Zhang, Jinyang, et al.
Published: (2025)

Towards Cross-Table Masked Pretraining for Web Data Mining
by: Ye, Chao, et al.
Published: (2023)

Pretrained Hybrids with MAD Skills
by: Roberts, Nicholas, et al.
Published: (2024)

Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings
by: Oguntade, Abass
Published: (2026)

Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation
by: Yu, Junjie, et al.
Published: (2026)

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
by: Wu, Jingfeng, et al.
Published: (2023)

Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
by: Wiedemer, Thaddäus, et al.
Published: (2025)

Learning to Forget with Information Divergence Reweighted Objectives for Noisy Labels
by: Birrell, Jeremiah, et al.
Published: (2025)

Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds
by: Shi, Jiaxin, et al.
Published: (2025)

Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing
by: Jain, Manish, et al.
Published: (2025)