Cover Image

SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lv, Changze, Li, Tianlong, Xu, Jianhan, Gu, Chenxi, Ling, Zixuan, Zhang, Cenyuan, Zheng, Xiaoqing, Huang, Xuanjing
Format:	Preprint
Published:	2023
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network
by: Lv, Changze, et al.
Published: (2023)

Spiking Convolutional Neural Networks for Text Classification
by: Lv, Changze, et al.
Published: (2024)

Decoding Continuous Character-based Language from Non-invasive Brain Recordings
by: Zhang, Cenyuan, et al.
Published: (2024)

Aligning Large Language Models with Human Preferences through Representation Engineering
by: Liu, Wenhao, et al.
Published: (2023)

Advancing Parameter Efficiency in Fine-tuning via Representation Editing
by: Wu, Muling, et al.
Published: (2024)

Promoting Data and Model Privacy in Federated Learning through Quantized LoRA
by: Zhu, JianHao, et al.
Published: (2024)

Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement
by: Wang, Xiaohua, et al.
Published: (2024)

Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling
by: Wang, Zhenghua, et al.
Published: (2025)

UPLex: Fine-Grained Personality Control in Large Language Models via Unsupervised Lexical Modulation
by: Li, Tianlong, et al.
Published: (2023)

Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective
by: Li, Tianlong, et al.
Published: (2024)

Improving Continual Pre-training Through Seamless Data Packing
by: Yin, Ruicheng, et al.
Published: (2025)

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
by: Zhang, Ying, et al.
Published: (2024)

Biologically Plausible Learning via Bidirectional Spike-Based Distillation
by: Lv, Changze, et al.
Published: (2025)

An investigation of structures responsible for gender bias in BERT and DistilBERT
by: Leteno, Thibaud, et al.
Published: (2024)

SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation
by: Bal, Malyaban, et al.
Published: (2023)

Leveraging IndoBERT and DistilBERT for Indonesian Emotion Classification in E-Commerce Reviews
by: Christian, William, et al.
Published: (2025)

A Knowledge-Enhanced Disease Diagnosis Method Based on Prompt Learning and BERT Integration
by: Zheng, Zhang
Published: (2024)

Efficient and Effective Time-Series Forecasting with Spiking Neural Networks
by: Lv, Changze, et al.
Published: (2024)

BERT-LID: Leveraging BERT to Improve Spoken Language Identification
by: Nie, Yuting, et al.
Published: (2022)

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators
by: Lv, Changze, et al.
Published: (2024)

SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT
by: Cho, Cheol Jun, et al.
Published: (2023)

CSSG: Measuring Code Similarity with Semantic Graphs
by: Lu, Yiyang, et al.
Published: (2026)

Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis
by: Wu, Yichao, et al.
Published: (2024)

Toward Relative Positional Encoding in Spiking Transformers
by: Lv, Changze, et al.
Published: (2025)

Enhancing TinyBERT for Financial Sentiment Analysis Using GPT-Augmented FinBERT Distillation
by: Thomas, Graison Jos
Published: (2024)

Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi
by: Raihan, Md Nishat, et al.
Published: (2023)

FaBERT: Pre-training BERT on Persian Blogs
by: Masumi, Mostafa, et al.
Published: (2024)

NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural
by: Wongso, Wilson, et al.
Published: (2024)

NeoBERT: A Next-Generation BERT
by: Breton, Lola Le, et al.
Published: (2025)

Progressive Mastery: Customized Curriculum Learning with Guided Prompting for Mathematical Reasoning
by: Wu, Muling, et al.
Published: (2025)

Weight-Inherited Distillation for Task-Agnostic BERT Compression
by: Wu, Taiqiang, et al.
Published: (2023)

AudioBERT: Audio Knowledge Augmented Language Model
by: Ok, Hyunjong, et al.
Published: (2024)

ACL: Aligned Contrastive Learning Improves BERT and Multi-exit BERT Fine-tuning
by: Li, Liz, et al.
Published: (2026)

NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew
by: Shmidman, Shaltiel, et al.
Published: (2025)

Benchmark^2: Systematic Evaluation of LLM Benchmarks
by: Qian, Qi, et al.
Published: (2026)

Best Practices for Distilling Large Language Models into BERT for Web Search Ranking
by: Ye, Dezhi, et al.
Published: (2024)

A Small and Fast BERT for Chinese Medical Punctuation Restoration
by: Ling, Tongtao, et al.
Published: (2023)

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?
by: Hościłowicz, Jakub, et al.
Published: (2023)

QiBERT -- Classifying Online Conversations Messages with BERT as a Feature
by: Ferreira-Saraiva, Bruno D., et al.
Published: (2024)

Pre-training technique to localize medical BERT and enhance biomedical BERT
by: Wada, Shoya, et al.
Published: (2020)