Cover Image

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Qingyang, Ke, Weimao
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Enhancing Question Answering Precision with Optimized Vector Retrieval and Instructions
by: Yang, Lixiao, et al.
Published: (2024)

DPad: Efficient Diffusion Language Models with Suffix Dropout
by: Chen, Xinhua, et al.
Published: (2025)

Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models
by: Cao, Jinghan, et al.
Published: (2026)

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP
by: Visser, Ruan, et al.
Published: (2026)

Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning
by: Huang, Jerry, et al.
Published: (2026)

TableTime: Reformulating Time Series Classification as Training-Free Table Understanding with Large Language Models
by: Wang, Jiahao, et al.
Published: (2024)

KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices
by: Zhou, Wuyang, et al.
Published: (2026)

SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
by: Joshi, Maithili, et al.
Published: (2025)

Negative to Positive Co-learning with Aggressive Modality Dropout
by: Magal, Nicholas, et al.
Published: (2025)

Baby Scale: Investigating Models Trained on Individual Children's Language Input
by: Feng, Steven Y., et al.
Published: (2026)

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
by: Xiao, Da, et al.
Published: (2025)

Investigating the Impact of Data Selection Strategies on Language Model Performance
by: Gu, Jiayao, et al.
Published: (2025)

Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization
by: Zeng, Yongcheng, et al.
Published: (2025)

Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali
by: Luitel, Nishant, et al.
Published: (2024)

Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
by: Zhang, Jun, et al.
Published: (2025)

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning
by: Xia, Wenhan, et al.
Published: (2024)

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
by: Saxena, Utkarsh, et al.
Published: (2024)

Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Models
by: Geiping, Jonas, et al.
Published: (2025)

Investigating Symbolic Capabilities of Large Language Models
by: Dave, Neisarg, et al.
Published: (2024)

Investigating Layer Importance in Large Language Models
by: Zhang, Yang, et al.
Published: (2024)

Cascade-Aware Training of Language Models
by: Wang, Congchao, et al.
Published: (2024)

Training Language Models to Reason Efficiently
by: Arora, Daman, et al.
Published: (2025)

TSO: Self-Training with Scaled Preference Optimization
by: Chen, Kaihui, et al.
Published: (2024)

BiasConnect: Investigating Bias Interactions in Text-to-Image Models
by: Shukla, Pushkar, et al.
Published: (2025)

A Survey on Training-free Alignment of Large Language Models
by: Pan, Birong, et al.
Published: (2025)

Asynchronous Local-SGD Training for Language Modeling
by: Liu, Bo, et al.
Published: (2024)

End-to-end Planner Training for Language Modeling
by: Cornille, Nathan, et al.
Published: (2024)

DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models
by: He, Wei, et al.
Published: (2024)

AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
by: Li, Zhiyuan, et al.
Published: (2026)

Mortgage Language Model: Domain-Adaptive Pretraining with Residual Instruction, Alignment Tuning, and Task-Specific Routing
by: Jain, Manish, et al.
Published: (2025)

Investigating Automatic Scoring and Feedback using Large Language Models
by: Katuka, Gloria Ashiya, et al.
Published: (2024)

Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
by: Xue, Huiyin, et al.
Published: (2025)

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
by: Li, Jiacheng, et al.
Published: (2026)

Residual Matrix Transformers: Scaling the Size of the Residual Stream
by: Mak, Brian, et al.
Published: (2025)

How to Train Long-Context Language Models (Effectively)
by: Gao, Tianyu, et al.
Published: (2024)

Direct Quantized Training of Language Models with Stochastic Rounding
by: Zhao, Kaiyan, et al.
Published: (2024)

Self-Training Large Language Models with Confident Reasoning
by: Jang, Hyosoon, et al.
Published: (2025)

Linear Dynamics in the RLVR Training of Large Language Models
by: Wang, Tianle, et al.
Published: (2026)

Language Model Training Paradigms for Clinical Feature Embeddings
by: Hu, Yurong, et al.
Published: (2023)

Training-Free Dynamic Upcycling of Expert Language Models
by: Fanì, Eros, et al.
Published: (2026)