How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912640708640768 |
|---|---|
| author | Inaba, Tatsuro Kamoda, Go Inui, Kentaro Isonuma, Masaru Miyao, Yusuke Oseki, Yohei Heinzerling, Benjamin Takagi, Yu |
| author_facet | Inaba, Tatsuro Kamoda, Go Inui, Kentaro Isonuma, Masaru Miyao, Yusuke Oseki, Yohei Heinzerling, Benjamin Takagi, Yu |
| contents | This study explores how bilingual language models develop complex internal representations. We employ sparse autoencoders to analyze internal representations of bilingual language models with a focus on the effects of training steps, layers, and model sizes. Our analysis shows that language models first learn languages separately, and then gradually form bilingual alignments, particularly in the mid layers. We also found that this bilingual tendency is stronger in larger models. Building on these findings, we demonstrate the critical role of bilingual representations in model performance by employing a novel method that integrates decomposed representations from a fully trained model into a mid-training model. Our results provide insights into how language models acquire bilingual capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06394 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders Inaba, Tatsuro Kamoda, Go Inui, Kentaro Isonuma, Masaru Miyao, Yusuke Oseki, Yohei Heinzerling, Benjamin Takagi, Yu Computation and Language Machine Learning This study explores how bilingual language models develop complex internal representations. We employ sparse autoencoders to analyze internal representations of bilingual language models with a focus on the effects of training steps, layers, and model sizes. Our analysis shows that language models first learn languages separately, and then gradually form bilingual alignments, particularly in the mid layers. We also found that this bilingual tendency is stronger in larger models. Building on these findings, we demonstrate the critical role of bilingual representations in model performance by employing a novel method that integrates decomposed representations from a fully trained model into a mid-training model. Our results provide insights into how language models acquire bilingual capabilities. |
| title | How a Bilingual LM Becomes Bilingual: Tracing Internal Representations with Sparse Autoencoders |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.06394 |