Improving Discrete Diffusion Models via Structured Preferential Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914814676172800 |
|---|---|
| author | Rissanen, Severi Heinonen, Markus Solin, Arno |
| author_facet | Rissanen, Severi Heinonen, Markus Solin, Arno |
| contents | In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17889 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Discrete Diffusion Models via Structured Preferential Generation Rissanen, Severi Heinonen, Markus Solin, Arno Machine Learning In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance. |
| title | Improving Discrete Diffusion Models via Structured Preferential Generation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.17889 |