Copertina

Understanding and Enhancing Mask-Based Pretraining towards Universal Representations

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Dong, Mingze, Wang, Leda, Kluger, Yuval
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps
di: Li, Henry, et al.
Pubblicazione: (2025)

Entropic Optimal Transport Eigenmaps for Nonlinear Alignment and Joint Embedding of High-Dimensional Datasets
di: Landa, Boris, et al.
Pubblicazione: (2024)

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling
di: Wang, Mingze, et al.
Pubblicazione: (2024)

Injecting Measurement Information Yields a Fast and Noise-Robust Diffusion-Based Inverse Problem Solver
di: Patsenker, Jonathan, et al.
Pubblicazione: (2025)

Euclidean Distance Deflation Under High-Dimensional Heteroskedastic Noise
di: Li, Keyi, et al.
Pubblicazione: (2025)

Noise Masking Attacks and Defenses for Pretrained Speech Models
di: Jagielski, Matthew, et al.
Pubblicazione: (2024)

Dual Diffusion for Unified Image Generation and Understanding
di: Li, Zijie, et al.
Pubblicazione: (2024)

Towards Cross-Table Masked Pretraining for Web Data Mining
di: Ye, Chao, et al.
Pubblicazione: (2023)

SpectralNet: Spectral Clustering using Deep Neural Networks
di: Shaham, Uri, et al.
Pubblicazione: (2018)

How Do the Architecture and Optimizer Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networks
di: Sharon, Yuval, et al.
Pubblicazione: (2024)

Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data
di: Chen, Jintai, et al.
Pubblicazione: (2024)

Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations
di: Slavutsky, Yuli, et al.
Pubblicazione: (2023)

Urban In-Context Learning: Bridging Pretraining and Inference through Masked Diffusion for Urban Profiling
di: Zhang, Ruixing, et al.
Pubblicazione: (2025)

Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection
di: Li, Ruilin, et al.
Pubblicazione: (2026)

Beyond Atoms: Enhancing Molecular Pretrained Representations with 3D Space Modeling
di: Lu, Shuqi, et al.
Pubblicazione: (2025)

CuPID: Leveraging Masked Single-Lead ECG Modelling for Enhancing the Representations
di: Atienza, Adtian, et al.
Pubblicazione: (2025)

On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
di: Wang, Mingze, et al.
Pubblicazione: (2025)

A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent
di: Wang, Mingze, et al.
Pubblicazione: (2023)

Bridging Foundation Models and Efficient Architectures: A Modular Brain Imaging Framework with Local Masking and Pretrained Representation Learning
di: Wang, Yanwen, et al.
Pubblicazione: (2025)

Towards Understanding Self-Pretraining for Sequence Classification
di: Coser, Omar, et al.
Pubblicazione: (2026)

Pretrained Visual Representations in Reinforcement Learning
di: Williams, Emlyn, et al.
Pubblicazione: (2024)

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
di: Zhang, Qi, et al.
Pubblicazione: (2024)

Meta-Prompt Optimization for LLM-Based Sequential Decision Making
di: Kong, Mingze, et al.
Pubblicazione: (2025)

Representation Deficiency in Masked Language Modeling
di: Meng, Yu, et al.
Pubblicazione: (2023)

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
di: Liu, Hongyi, et al.
Pubblicazione: (2025)

KCM: KAN-Based Collaboration Models Enhance Pretrained Large Models
di: Dai, Guangyu, et al.
Pubblicazione: (2025)

SleepNet and DreamNet: Enriching and Reconstructing Representations for Consolidated Visual Classification
di: Ni, Mingze, et al.
Pubblicazione: (2024)

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
di: Zheng, Bo, et al.
Pubblicazione: (2026)

Mask the Redundancy: Evolving Masking Representation Learning for Multivariate Time-Series Clustering
di: Tan, Zexi, et al.
Pubblicazione: (2025)

Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals
di: Liu, Ran, et al.
Pubblicazione: (2023)

Understanding Differential Transformer Unchains Pretrained Self-Attentions
di: Kong, Chaerin, et al.
Pubblicazione: (2025)

Myna: Masking-Based Contrastive Learning of Musical Representations
di: Yonay, Ori, et al.
Pubblicazione: (2025)

Value Explicit Pretraining for Learning Transferable Representations
di: Lekkala, Kiran, et al.
Pubblicazione: (2023)

Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations
di: Alsamkary, Hazem, et al.
Pubblicazione: (2025)

Structure-Aware Masking for Protein Representation Learning
di: Walton, Thomas, et al.
Pubblicazione: (2026)

United We Pretrain, Divided We Fail! Representation Learning for Time Series by Pretraining on 75 Datasets at Once
di: Kraus, Maurice, et al.
Pubblicazione: (2024)

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
di: Messmer, Bettina, et al.
Pubblicazione: (2025)

DailyMAE: Towards Pretraining Masked Autoencoders in One Day
di: Wu, Jiantao, et al.
Pubblicazione: (2024)

Towards Universal Neural Operators through Multiphysics Pretraining
di: Masliaev, Mikhail, et al.
Pubblicazione: (2025)

Enhancing TCR-Peptide Interaction Prediction with Pretrained Language Models and Molecular Representations
di: Qi, Cong, et al.
Pubblicazione: (2025)