Copertina

Enhancing Training Efficiency Using Packing with Flash Attention

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Kundu, Achintya, Lee, Rhui Dih, Wynter, Laura, Ganti, Raghu Kiran, Mishra, Mayank
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
di: Lee, Rhui Dih, et al.
Pubblicazione: (2024)

Collaboratively adding new knowledge to an LLM
di: Lee, Rhui Dih, et al.
Pubblicazione: (2024)

Efficiently Distilling LLMs for Edge Applications
di: Kundu, Achintya, et al.
Pubblicazione: (2024)

INT-FlashAttention: Enabling Flash Attention for INT8 Quantization
di: Chen, Shimao, et al.
Pubblicazione: (2024)

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
di: Qiu, Haiquan, et al.
Pubblicazione: (2025)

Flash Invariant Point Attention
di: Liu, Andrew, et al.
Pubblicazione: (2025)

AMLA: MUL by ADD in FlashAttention Rescaling
di: Liao, Qichen, et al.
Pubblicazione: (2025)

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
di: Shah, Jay, et al.
Pubblicazione: (2024)

FlashOptim: Optimizers for Memory-Efficient Training
di: Ortiz, Jose Javier Gonzalez, et al.
Pubblicazione: (2026)

Is In-Context Learning Learning?
di: de Wynter, Adrian
Pubblicazione: (2025)

One-shot World Models Using a Transformer Trained on a Synthetic Prior
di: Ferreira, Fabio, et al.
Pubblicazione: (2024)

DELPHYNE: A Pre-Trained Model for General and Financial Time Series
di: Ding, Xueying, et al.
Pubblicazione: (2025)

Enhancing Performance and Scalability of Large-Scale Recommendation Systems with Jagged Flash Attention
di: Xu, Rengan, et al.
Pubblicazione: (2024)

Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
di: Gupta, Sonam, et al.
Pubblicazione: (2024)

Efficiently Dispatching Flash Attention For Partially Filled Attention Masks
di: Sharma, Agniv, et al.
Pubblicazione: (2024)

Efficiency Boost in Decentralized Optimization: Reimagining Neighborhood Aggregation with Minimal Overhead
di: Kalwar, Durgesh, et al.
Pubblicazione: (2025)

An Algorithm for Learning Smaller Representations of Models With Scarce Data
di: de Wynter, Adrian
Pubblicazione: (2020)

VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation
di: Sun, Yupeng, et al.
Pubblicazione: (2026)

VarDrop: Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting
di: Kang, Junhyeok, et al.
Pubblicazione: (2025)

Enhancing Data Efficiency and Feature Identification for Lithium-Ion Battery Lifespan Prediction by Deciphering Interpretation of Temporal Patterns and Cyclic Variability Using Attention-Based Models
di: Lee, Jaewook, et al.
Pubblicazione: (2023)

Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
di: Lu, Han, et al.
Pubblicazione: (2025)

FLASH-D: FlashAttention with Hidden Softmax Division
di: Alexandridis, Kosmas, et al.
Pubblicazione: (2025)

Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels
di: Beck, Maximilian, et al.
Pubblicazione: (2025)

QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
di: Oh, Sehyeon, et al.
Pubblicazione: (2026)

FlashOmni: A Unified Sparse Attention Engine for Diffusion Transformers
di: Qiao, Liang, et al.
Pubblicazione: (2025)

Sawtooth Wavefront Reordering: Enhanced CuTile FlashAttention on NVIDIA GB10
di: Zhu, Yifan, et al.
Pubblicazione: (2026)

LEO: Graph Attention Network based Hybrid Multi Sensor Extended Object Fusion and Tracking for Autonomous Driving Applications
di: Mayank, Mayank, et al.
Pubblicazione: (2026)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
di: Pan, Bowen, et al.
Pubblicazione: (2024)

M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
di: Mishra, Mayank, et al.
Pubblicazione: (2026)

SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
di: Guo, Wentao, et al.
Pubblicazione: (2025)

GFPack++: Improving 2D Irregular Packing by Learning Gradient Field with Attention
di: Xue, Tianyang, et al.
Pubblicazione: (2024)

How Good Are LLMs at Processing Tool Outputs?
di: Kate, Kiran, et al.
Pubblicazione: (2025)

Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
di: Joshi, Thomas, et al.
Pubblicazione: (2025)

Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization
di: Kim, Dongjun, et al.
Pubblicazione: (2026)

Flash STU: Fast Spectral Transform Units
di: Liu, Y. Isabel, et al.
Pubblicazione: (2024)

Throughput Optimization as a Strategic Lever in Large-Scale AI Systems: Evidence from Dataloader and Memory Profiling Innovations
di: Jha, Mayank
Pubblicazione: (2026)

A Real-time Anomaly Detection Using Convolutional Autoencoder with Dynamic Threshold
di: Maitra, Sarit, et al.
Pubblicazione: (2024)

DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
di: Rangwani, Harsh, et al.
Pubblicazione: (2024)

Monodense Deep Neural Model for Determining Item Price Elasticity
di: Garg, Lakshya, et al.
Pubblicazione: (2026)

Leveraging Data Symmetries to Select an Optimal Subset of Training Data under Label Noise
di: Shubham, Kumar, et al.
Pubblicazione: (2026)