Copertina

GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Javadi, Farnoosh, Ahmed, Walid, Hajimolahoseini, Habib, Ataiefard, Foozhan, Hassanpour, Mohammad, Asani, Saina, Wen, Austin, Awad, Omar Mohamed, Liu, Kangling, Liu, Yang
Natura:	Preprint
Pubblicazione:	2023
Soggetti:	Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection
di: Ataiefard, Foozhan, et al.
Pubblicazione: (2024)

Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model
di: Hajimolahoseini, Habib, et al.
Pubblicazione: (2024)

Accelerating the Low-Rank Decomposed Models
di: Hajimolahoseini, Habib, et al.
Pubblicazione: (2024)

Is 3D Convolution with 5D Tensors Really Necessary for Video Analysis?
di: Hajimolahoseini, Habib, et al.
Pubblicazione: (2024)

Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization
di: Hajimolahoseini, Habib, et al.
Pubblicazione: (2023)

Improving Resnet-9 Generalization Trained on Small Datasets
di: Awad, Omar Mohamed, et al.
Pubblicazione: (2023)

Defense That Attacks: How Robust Models Become Better Attackers
di: Awad, Mohamed, et al.
Pubblicazione: (2025)

ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training
di: Dialameh, Maryam, et al.
Pubblicazione: (2025)

Edit Distance of Finite-Valued Transducers
di: Mathew, Prince, et al.
Pubblicazione: (2026)

Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
di: Lin, Zhenghao, et al.
Pubblicazione: (2025)

Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention
di: Khan, Zohaib, et al.
Pubblicazione: (2024)

A Novel Approach to Translate Structural Aggregation Queries to MapReduce Code
di: Abdelmoniem, Ahmed M., et al.
Pubblicazione: (2025)

Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training
di: Qin, Yiwei, et al.
Pubblicazione: (2026)

MLAR: Multi-layer Large Language Model-based Robotic Process Automation Applicant Tracking
di: Younes, Mohamed T., et al.
Pubblicazione: (2025)

CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
di: Karim, Rezaul, et al.
Pubblicazione: (2026)

Event Camera Data Dense Pre-training
di: Yang, Yan, et al.
Pubblicazione: (2023)

Classification of Diabetic Retinopathy using Pre-Trained Deep Learning Models
di: Al-Kamachy, Inas, et al.
Pubblicazione: (2024)

On Predicting the Post-training Potential of Pre-trained LLMs
di: Li, Xiaoyuan, et al.
Pubblicazione: (2026)

MonoMSK: Monocular 3D Musculoskeletal Dynamics Estimation
di: Koleini, Farnoosh, et al.
Pubblicazione: (2025)

Beyond Fixed Length: Bucket Pre-training is All You Need
di: Yang, Qing, et al.
Pubblicazione: (2024)

Augmented Fine-Tuned LLMs for Enhanced Recruitment Automation
di: Younes, Mohamed T., et al.
Pubblicazione: (2025)

MSLEF: Multi-Segment LLM Ensemble Finetuning in Recruitment
di: Walid, Omar, et al.
Pubblicazione: (2025)

ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time
di: Zahirnia, Kiarash, et al.
Pubblicazione: (2025)

Exploring the Efficacy of Group-Normalization in Deep Learning Models for Alzheimer's Disease Classification
di: Habib, Gousia, et al.
Pubblicazione: (2024)

Activation-aware Probe-Query: Effective Key-Value Retrieval for Long-Context LLMs Inference
di: Xiao, Qingfa, et al.
Pubblicazione: (2025)

Probing Language Models for Pre-training Data Detection
di: Liu, Zhenhua, et al.
Pubblicazione: (2024)

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
di: Deng, Zihao, et al.
Pubblicazione: (2023)

Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training
di: Peng, Jiahui, et al.
Pubblicazione: (2025)

Contrastive Language Video Time Pre-training
di: Liu, Hengyue, et al.
Pubblicazione: (2024)

Text Grouping Adapter: Adapting Pre-trained Text Detector for Layout Analysis
di: Bi, Tianci, et al.
Pubblicazione: (2024)

CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding
di: Abdelrahman, Eslam, et al.
Pubblicazione: (2023)

On Pre-training of Multimodal Language Models Customized for Chart Understanding
di: Fan, Wan-Cyuan, et al.
Pubblicazione: (2024)

In Pursuit of Pixel Supervision for Visual Pre-training
di: Yang, Lihe, et al.
Pubblicazione: (2025)

GLID: Pre-training a Generalist Encoder-Decoder Vision Model
di: Liu, Jihao, et al.
Pubblicazione: (2024)

Visually Guided Generative Text-Layout Pre-training for Document Intelligence
di: Mao, Zhiming, et al.
Pubblicazione: (2024)

Ecstasy and Enlightenment
di: Asani, Ali S.
Pubblicazione: (2025)

3D Scene Graph Guided Vision-Language Pre-training
di: Liu, Hao, et al.
Pubblicazione: (2024)

Masked Clustering Prediction for Unsupervised Point Cloud Pre-training
di: Ren, Bin, et al.
Pubblicazione: (2025)

Fast and Accurate Bayesian Optimization with Pre-trained Transformers for Constrained Engineering Problems
di: Rosen, et al.
Pubblicazione: (2024)

How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
di: Lu, Xin, et al.
Pubblicazione: (2025)