Cover Image

Symmetric Dot-Product Attention for Efficient Training of BERT Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Courtois, Martin, Ostendorff, Malte, Hennig, Leonhard, Rehm, Georg
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Investigating Gender Bias in Turkish Language Models
by: Caglidil, Orhun Mersin, et al.
Published: (2024)

Reward Modeling with Weak Supervision for Language Models
by: Hauptvogel, Ben, et al.
Published: (2024)

HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information
by: Ruan, Qian, et al.
Published: (2022)

Entity Linking using LLMs for Automated Product Carbon Footprint Estimation
by: Castle, Steffen, et al.
Published: (2025)

Multilingual European Language Models: Benchmarking Approaches and Challenges
by: Barth, Fabio, et al.
Published: (2025)

Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
by: Borisova, Ekaterina, et al.
Published: (2025)

SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing
by: Foppiano, Luca, et al.
Published: (2025)

Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?
by: Rehm, Georg, et al.
Published: (2025)

PyTorch-IE: Fast and Reproducible Prototyping for Information Extraction
by: Binder, Arne, et al.
Published: (2024)

Positional Attention for Efficient BERT-Based Named Entity Recognition
by: Sun, Mo, et al.
Published: (2025)

UniBERT: Adversarial Training for Language-Universal Representations
by: Avram, Andrei-Marius, et al.
Published: (2025)

LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations
by: Wang, Qianli, et al.
Published: (2024)

Analyzing Multi-Head Attention on Trojan BERT Models
by: Wang, Jingwei
Published: (2024)

SwissBERT: The Multilingual Language Model for Switzerland
by: Vamvas, Jannis, et al.
Published: (2023)

How desirable is alignment between LLMs and linguistically diverse human users?
by: Knoeferle, Pia, et al.
Published: (2025)

Sliding Window Attention Training for Efficient Large Language Models
by: Fu, Zichuan, et al.
Published: (2025)

KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP
by: Metinov, Adilet, et al.
Published: (2025)

GottBERT: a pure German Language Model
by: Scheible, Raphael, et al.
Published: (2020)

ConfliBERT: A Language Model for Political Conflict
by: Brandt, Patrick T., et al.
Published: (2024)

Extending Translate-Train for ColBERT-X to African Language CLIR
by: Yang, Eugene, et al.
Published: (2024)

SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation
by: Lv, Changze, et al.
Published: (2023)

ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims
by: Ahmad, Raia Abu, et al.
Published: (2026)

Analysis of Argument Structure Constructions in the Large Language Model BERT
by: Ramezani, Pegah, et al.
Published: (2024)

Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax
by: Zaitova, Iuliia, et al.
Published: (2025)

Training-free Context-adaptive Attention for Efficient Long Context Modeling
by: You, Zeng, et al.
Published: (2025)

Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition
by: Shirke, Mayur, et al.
Published: (2025)

ManufactuBERT: Efficient Continual Pretraining for Manufacturing
by: Armingaud, Robin, et al.
Published: (2025)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

Detecting Bias in Large Language Models: Fine-tuned KcBERT
by: Lee, J. K., et al.
Published: (2024)

CultureBERT: Measuring Corporate Culture With Transformer-Based Language Models
by: Koch, Sebastian, et al.
Published: (2022)

Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition
by: Piñeiro-Martín, Andrés, et al.
Published: (2024)

MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain
by: Bressem, Keno K., et al.
Published: (2023)

ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reports
by: Yamagishi, Yosuke, et al.
Published: (2025)

An Encoder-Integrated PhoBERT with Graph Attention for Vietnamese Token-Level Classification
by: Nguyen, Ba-Quang
Published: (2025)

Training BERT Models to Carry Over a Coding System Developed on One Corpus to Another
by: Galambos, Dalma, et al.
Published: (2023)

BERT-LSH: Reducing Absolute Compute For Attention
by: Li, Zezheng, et al.
Published: (2024)

MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling
by: Kim, Jinwoong, et al.
Published: (2026)

MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language
by: Yadav, Sumit, et al.
Published: (2025)

EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora
by: Qarah, Faisal
Published: (2024)

From BERT to LLMs: Comparing and Understanding Chinese Classifier Prediction in Language Models
by: Zhang, Ziqi, et al.
Published: (2025)