DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914545607376896 |
|---|---|
| author | Wang, Jingyuan Jia, Zhihao Liu, Chenyu Zhou, Xinliang Luo, Haoran Jia, Ziyu Li, Yong Li, Fang Yao, Junfeng Ding, Yi |
| author_facet | Wang, Jingyuan Jia, Zhihao Liu, Chenyu Zhou, Xinliang Luo, Haoran Jia, Ziyu Li, Yong Li, Fang Yao, Junfeng Ding, Yi |
| contents | EEG foundation models (EFMs) achieve strong cross-subject and cross-task generalization through large-scale pretraining and downstream fine-tuning. Through empirical analysis, we observe that (i) task-adapted EFMs provide strong decoding performance but incur substantial overhead when retained as inference backbones, making knowledge distillation a natural route for optimizing compact students; and (ii) direct distillation from a fixed teacher representation underutilizes EFM knowledge, as task-discriminative information is distributed across intermediate layers rather than concentrated in the final layer. These observations motivate DLink (Distilling Layer-wise and Dominant Knowledge), a spectrally guided distillation framework with input-conditioned layer routing for transferring EFM knowledge into compact students. DLink uses a lightweight router to aggregate teacher layers for each input, and aligns magnitude and phase spectra to mitigate compression-induced spectral distortion in learned representations. The routed teacher knowledge is internalized by a project-then-compress student; the teacher and router are used only during training. Experiments on four EEG benchmarks show that DLink improves matched compact students and remains competitive with lightweight baselines, narrowing the gap to fine-tuned EFMs while substantially reducing parameters, FLOPs, and CPU-only inference latency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_15016 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models Wang, Jingyuan Jia, Zhihao Liu, Chenyu Zhou, Xinliang Luo, Haoran Jia, Ziyu Li, Yong Li, Fang Yao, Junfeng Ding, Yi Machine Learning EEG foundation models (EFMs) achieve strong cross-subject and cross-task generalization through large-scale pretraining and downstream fine-tuning. Through empirical analysis, we observe that (i) task-adapted EFMs provide strong decoding performance but incur substantial overhead when retained as inference backbones, making knowledge distillation a natural route for optimizing compact students; and (ii) direct distillation from a fixed teacher representation underutilizes EFM knowledge, as task-discriminative information is distributed across intermediate layers rather than concentrated in the final layer. These observations motivate DLink (Distilling Layer-wise and Dominant Knowledge), a spectrally guided distillation framework with input-conditioned layer routing for transferring EFM knowledge into compact students. DLink uses a lightweight router to aggregate teacher layers for each input, and aligns magnitude and phase spectra to mitigate compression-induced spectral distortion in learned representations. The routed teacher knowledge is internalized by a project-then-compress student; the teacher and router are used only during training. Experiments on four EEG benchmarks show that DLink improves matched compact students and remains competitive with lightweight baselines, narrowing the gap to fine-tuned EFMs while substantially reducing parameters, FLOPs, and CPU-only inference latency. |
| title | DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.15016 |