From Words to Amino Acids: Does the Curse of Depth Persist?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915954033688576 |
|---|---|
| author | Siji, Aleena Mamaghan, Amir Mohammad Karimi Kapl, Ferdinand Höppe, Tobias Angelis, Emmanouil Dittadi, Andrea Brenner, Maurice Heinzinger, Michael Johansson, Karl Henrik Maile, Kaitlin von Oswald, Johannes Bauer, Stefan |
| author_facet | Siji, Aleena Mamaghan, Amir Mohammad Karimi Kapl, Ferdinand Höppe, Tobias Angelis, Emmanouil Dittadi, Andrea Brenner, Maurice Heinzinger, Michael Johansson, Karl Henrik Maile, Kaitlin von Oswald, Johannes Bauer, Stefan |
| contents | Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language models (LLMs), they are typically trained as deep transformers with next-token or masked-token prediction objectives on massive sequence corpora and are scaled by increasing model depth. Recent work on autoregressive LLMs has identified the Curse of Depth: many later layers contribute little to the final output predictions. These findings naturally raise the question of whether a similar depth inefficiency also appears in PLMs, where many widely used models are not autoregressive, and some are multimodal, accepting both protein sequence and structure as input. In this work, we present a depth analysis of seven popular PLM families across model scales, spanning autoregressive, masked, and diffusion objectives, and quantify how layer contributions evolve with depth using a unified set of probing-, perturbation-, and downstream-evaluation measurements. Across models, we observe consistent depth-dependent patterns that extend prior findings on LLMs: a large fraction of task-relevant computation is concentrated in a subset of layers, while the remaining layers mainly provide incremental refinement of the final prediction. These trends persist beyond sequence-only settings and also appear in multimodal PLMs. Taken together, our results suggest that depth inefficiency is a common feature of modern PLMs, motivating future work on more depth-efficient architectures and training methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_21750 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Words to Amino Acids: Does the Curse of Depth Persist? Siji, Aleena Mamaghan, Amir Mohammad Karimi Kapl, Ferdinand Höppe, Tobias Angelis, Emmanouil Dittadi, Andrea Brenner, Maurice Heinzinger, Michael Johansson, Karl Henrik Maile, Kaitlin von Oswald, Johannes Bauer, Stefan Machine Learning Protein language models (PLMs) have become widely adopted as general-purpose models, demonstrating strong performance in protein engineering and de novo design. Like large language models (LLMs), they are typically trained as deep transformers with next-token or masked-token prediction objectives on massive sequence corpora and are scaled by increasing model depth. Recent work on autoregressive LLMs has identified the Curse of Depth: many later layers contribute little to the final output predictions. These findings naturally raise the question of whether a similar depth inefficiency also appears in PLMs, where many widely used models are not autoregressive, and some are multimodal, accepting both protein sequence and structure as input. In this work, we present a depth analysis of seven popular PLM families across model scales, spanning autoregressive, masked, and diffusion objectives, and quantify how layer contributions evolve with depth using a unified set of probing-, perturbation-, and downstream-evaluation measurements. Across models, we observe consistent depth-dependent patterns that extend prior findings on LLMs: a large fraction of task-relevant computation is concentrated in a subset of layers, while the remaining layers mainly provide incremental refinement of the final prediction. These trends persist beyond sequence-only settings and also appear in multimodal PLMs. Taken together, our results suggest that depth inefficiency is a common feature of modern PLMs, motivating future work on more depth-efficient architectures and training methods. |
| title | From Words to Amino Acids: Does the Curse of Depth Persist? |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.21750 |