Conditional [MASK] Discrete Diffusion Language Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912243366494208 |
|---|---|
| author | Koh, Hyukhun Jhang, Minha Kim, Dohyung Lee, Sangmook Jung, Kyomin |
| author_facet | Koh, Hyukhun Jhang, Minha Kim, Dohyung Lee, Sangmook Jung, Kyomin |
| contents | Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropy-based noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06438 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Conditional [MASK] Discrete Diffusion Language Model Koh, Hyukhun Jhang, Minha Kim, Dohyung Lee, Sangmook Jung, Kyomin Computation and Language Artificial Intelligence Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropy-based noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation. |
| title | Conditional [MASK] Discrete Diffusion Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.06438 |