Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917986590261248 |
|---|---|
| author | Tong, Anh Nguyen-Tang, Thanh Lee, Dongeun Nguyen, Duc Tran, Toan Hall, David Kang, Cheongwoong Choi, Jaesik |
| author_facet | Tong, Anh Nguyen-Tang, Thanh Lee, Dongeun Nguyen, Duc Tran, Toan Hall, David Kang, Cheongwoong Choi, Jaesik |
| contents | Recent advancements in large language models (LLMs) based on transformer architectures have sparked significant interest in understanding their inner workings. In this paper, we introduce a novel approach to modeling transformer architectures using highly flexible non-autonomous neural ordinary differential equations (ODEs). Our proposed model parameterizes all weights of attention and feed-forward blocks through neural networks, expressing these weights as functions of a continuous layer index. Through spectral analysis of the model's dynamics, we uncover an increase in eigenvalue magnitude that challenges the weight-sharing assumption prevalent in existing theoretical studies. We also leverage the Lyapunov exponent to examine token-level sensitivity, enhancing model interpretability. Our neural ODE transformer demonstrates performance comparable to or better than vanilla transformers across various configurations and datasets, while offering flexible fine-tuning capabilities that can adapt to different architectural constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01329 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning Tong, Anh Nguyen-Tang, Thanh Lee, Dongeun Nguyen, Duc Tran, Toan Hall, David Kang, Cheongwoong Choi, Jaesik Machine Learning Artificial Intelligence Recent advancements in large language models (LLMs) based on transformer architectures have sparked significant interest in understanding their inner workings. In this paper, we introduce a novel approach to modeling transformer architectures using highly flexible non-autonomous neural ordinary differential equations (ODEs). Our proposed model parameterizes all weights of attention and feed-forward blocks through neural networks, expressing these weights as functions of a continuous layer index. Through spectral analysis of the model's dynamics, we uncover an increase in eigenvalue magnitude that challenges the weight-sharing assumption prevalent in existing theoretical studies. We also leverage the Lyapunov exponent to examine token-level sensitivity, enhancing model interpretability. Our neural ODE transformer demonstrates performance comparable to or better than vanilla transformers across various configurations and datasets, while offering flexible fine-tuning capabilities that can adapt to different architectural constraints. |
| title | Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.01329 |