PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910378809622528 |
|---|---|
| author | Zhang, Yizhe Gu, Jiatao Wu, Zhuofeng Zhai, Shuangfei Susskind, Josh Jaitly, Navdeep |
| author_facet | Zhang, Yizhe Gu, Jiatao Wu, Zhuofeng Zhai, Shuangfei Susskind, Josh Jaitly, Navdeep |
| contents | Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to exposure bias - the difference between how a model is trained, and how it is used during inference. Denoising diffusion models provide an alternative approach in which a model can revisit and revise its output. However, they can be computationally expensive and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module that uses latent diffusion to generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed method is evaluated on various conditional generation tasks, and results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text in an efficient manner. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_02531 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model Zhang, Yizhe Gu, Jiatao Wu, Zhuofeng Zhai, Shuangfei Susskind, Josh Jaitly, Navdeep Computation and Language Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to exposure bias - the difference between how a model is trained, and how it is used during inference. Denoising diffusion models provide an alternative approach in which a model can revisit and revise its output. However, they can be computationally expensive and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module that uses latent diffusion to generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed method is evaluated on various conditional generation tasks, and results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text in an efficient manner. |
| title | PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2306.02531 |