Copertina

DEPT: Decoupled Embeddings for Pre-training Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Iacob, Alex, Sani, Lorenzo, Kurmanji, Meghdad, Shen, William F., Qiu, Xinchi, Cai, Dongqi, Gao, Yan, Lane, Nicholas D.
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Response-Conditioned Parallel-to-Sequential Orchestration for Multi-Agent Systems
di: Tastan, Nurbek, et al.
Pubblicazione: (2026)

LLM Unlearning via Neural Activation Redirection
di: Shen, William F., et al.
Pubblicazione: (2025)

MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
di: Iacob, Alex, et al.
Pubblicazione: (2025)

How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspective
di: Qiu, Xinchi, et al.
Pubblicazione: (2024)

Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?
di: Li, Zexi, et al.
Pubblicazione: (2025)

The Future of Large Language Model Pre-training is Federated
di: Sani, Lorenzo, et al.
Pubblicazione: (2024)

AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling
di: Aleksandrov, Preslav, et al.
Pubblicazione: (2025)

DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
di: Iacob, Alex, et al.
Pubblicazione: (2025)

Photon: Federated LLM Pre-Training
di: Sani, Lorenzo, et al.
Pubblicazione: (2024)

Worldwide Federated Training of Language Models
di: Iacob, Alex, et al.
Pubblicazione: (2024)

Position: Bridge the Gaps between Machine Unlearning and AI Regulation
di: Marino, Bill, et al.
Pubblicazione: (2025)

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
di: Jovanović, Andrej, et al.
Pubblicazione: (2026)

Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages
di: Zhao, Wanru, et al.
Pubblicazione: (2025)

FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clients
di: Qiu, Xinchi, et al.
Pubblicazione: (2024)

Pollen: High-throughput Federated Learning Simulation via Resource-Aware Client Placement
di: Sani, Lorenzo, et al.
Pubblicazione: (2023)

Sheaf HyperNetworks for Personalized Federated Learning
di: Nguyen, Bao, et al.
Pubblicazione: (2024)

$f$-FUM: Federated Unlearning via min--max and $f$-divergence
di: Karimian, Radmehr, et al.
Pubblicazione: (2026)

SparsyFed: Sparse Adaptive Federated Training
di: Guastella, Adriano, et al.
Pubblicazione: (2025)

Task-Centric Personalized Federated Fine-Tuning of Language Models
di: Talasso, Gabriel U., et al.
Pubblicazione: (2026)

KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
di: Li, Yudong, et al.
Pubblicazione: (2026)

SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstention
di: Shen, William F., et al.
Pubblicazione: (2025)

VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
di: Xiao, Wenyi, et al.
Pubblicazione: (2026)

Stable Language Model Pre-training by Reducing Embedding Variability
di: Chung, Woojin, et al.
Pubblicazione: (2024)

Small Language Models: Survey, Measurements, and Insights
di: Lu, Zhenyan, et al.
Pubblicazione: (2024)

Decoupled DiLoCo for Resilient Distributed Pre-training
di: Douillard, Arthur, et al.
Pubblicazione: (2026)

Prompting Disentangled Embeddings for Knowledge Graph Completion with Pre-trained Language Model
di: Geng, Yuxia, et al.
Pubblicazione: (2023)

On Initializing Transformers with Pre-trained Embeddings
di: Kim, Ha Young, et al.
Pubblicazione: (2024)

Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models
di: Zhang, Ying, et al.
Pubblicazione: (2024)

Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings
di: Sharma, Kartik, et al.
Pubblicazione: (2025)

ARS: Adaptive Reasoning Suppression for Efficient Large Reasoning Language Models
di: Zheng, Dongqi
Pubblicazione: (2025)

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
di: Zheng, Junhao, et al.
Pubblicazione: (2023)

Metadata Conditioning Accelerates Language Model Pre-training
di: Gao, Tianyu, et al.
Pubblicazione: (2025)

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
di: Jin, Yang, et al.
Pubblicazione: (2024)

Text Embeddings by Weakly-Supervised Contrastive Pre-training
di: Wang, Liang, et al.
Pubblicazione: (2022)

Anatomical Structure-Guided Medical Vision-Language Pre-training
di: Li, Qingqiu, et al.
Pubblicazione: (2024)

Investigating the Impact of Language-Adaptive Fine-Tuning on Sentiment Analysis in Hausa Language Using AfriBERTa
di: Sani, Sani Abdullahi, et al.
Pubblicazione: (2025)

Model Merging in Pre-training of Large Language Models
di: Li, Yunshui, et al.
Pubblicazione: (2025)

Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization
di: Zhang, Haode, et al.
Pubblicazione: (2022)

Making Pre-trained Language Models Great on Tabular Prediction
di: Yan, Jiahuan, et al.
Pubblicazione: (2024)

Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
di: Chen, Shan, et al.
Pubblicazione: (2024)