Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917350172786688 |
|---|---|
| author | Kishino, Ryo Shiomi, Riku Yamagiwa, Hiroaki Oyama, Momose Shimodaira, Hidetoshi |
| author_facet | Kishino, Ryo Shiomi, Riku Yamagiwa, Hiroaki Oyama, Momose Shimodaira, Hidetoshi |
| contents | Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence to the target model compared with uniform weighting over the Pile. Although knowledge distillation remains more effective when available, the proposed method still achieves meaningful alignment, and downstream task performance also tends to become closer to that of the target model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_16622 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model Kishino, Ryo Shiomi, Riku Yamagiwa, Hiroaki Oyama, Momose Shimodaira, Hidetoshi Computation and Language Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence to the target model compared with uniform weighting over the Pile. Although knowledge distillation remains more effective when available, the proposed method still achieves meaningful alignment, and downstream task performance also tends to become closer to that of the target model. |
| title | Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.16622 |