Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.08379 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913238170468352 |
|---|---|
| author | Mahabadi, Rabeeh Karimi Ivison, Hamish Tae, Jaesung Henderson, James Beltagy, Iz Peters, Matthew E. Cohan, Arman |
| author_facet | Mahabadi, Rabeeh Karimi Ivison, Hamish Tae, Jaesung Henderson, James Beltagy, Iz Peters, Matthew E. Cohan, Arman |
| contents | Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various continuous domains. However, applying continuous diffusion models to natural language remains challenging due to its discrete nature and the need for a large number of diffusion steps to generate text, making diffusion-based generation expensive. In this work, we propose Text-to-text Self-conditioned Simplex Diffusion (TESS), a text diffusion model that is fully non-autoregressive, employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the learned embedding space. Through extensive experiments on natural language understanding and generation tasks including summarization, text simplification, paraphrase generation, and question generation, we demonstrate that TESS outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. We publicly release our codebase at https://github.com/allenai/tess-diffusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_08379 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | TESS: Text-to-Text Self-Conditioned Simplex Diffusion Mahabadi, Rabeeh Karimi Ivison, Hamish Tae, Jaesung Henderson, James Beltagy, Iz Peters, Matthew E. Cohan, Arman Computation and Language Machine Learning Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various continuous domains. However, applying continuous diffusion models to natural language remains challenging due to its discrete nature and the need for a large number of diffusion steps to generate text, making diffusion-based generation expensive. In this work, we propose Text-to-text Self-conditioned Simplex Diffusion (TESS), a text diffusion model that is fully non-autoregressive, employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the learned embedding space. Through extensive experiments on natural language understanding and generation tasks including summarization, text simplification, paraphrase generation, and question generation, we demonstrate that TESS outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. We publicly release our codebase at https://github.com/allenai/tess-diffusion. |
| title | TESS: Text-to-Text Self-Conditioned Simplex Diffusion |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2305.08379 |