Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915021646200832 |
|---|---|
| author | Vouitsis, Noël Hosseinzadeh, Rasa Ross, Brendan Leigh Villecroze, Valentin Gorti, Satya Krishna Cresswell, Jesse C. Loaiza-Ganem, Gabriel |
| author_facet | Vouitsis, Noël Hosseinzadeh, Rasa Ross, Brendan Leigh Villecroze, Valentin Gorti, Satya Krishna Cresswell, Jesse C. Loaiza-Ganem, Gabriel |
| contents | Although diffusion models can generate remarkably high-quality samples, they are intrinsically bottlenecked by their expensive iterative sampling procedure. Consistency models (CMs) have recently emerged as a promising diffusion model distillation method, reducing the cost of sampling by generating high-fidelity samples in just a few iterations. Consistency model distillation aims to solve the probability flow ordinary differential equation (ODE) defined by an existing diffusion model. CMs are not directly trained to minimize error against an ODE solver, rather they use a more computationally tractable objective. As a way to study how effectively CMs solve the probability flow ODE, and the effect that any induced error has on the quality of generated samples, we introduce Direct CMs, which \textit{directly} minimize this error. Intriguingly, we find that Direct CMs reduce the ODE solving error compared to CMs but also result in significantly worse sample quality, calling into question why exactly CMs work well in the first place. Full code is available at: https://github.com/layer6ai-labs/direct-cms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_08954 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples Vouitsis, Noël Hosseinzadeh, Rasa Ross, Brendan Leigh Villecroze, Valentin Gorti, Satya Krishna Cresswell, Jesse C. Loaiza-Ganem, Gabriel Machine Learning Artificial Intelligence Although diffusion models can generate remarkably high-quality samples, they are intrinsically bottlenecked by their expensive iterative sampling procedure. Consistency models (CMs) have recently emerged as a promising diffusion model distillation method, reducing the cost of sampling by generating high-fidelity samples in just a few iterations. Consistency model distillation aims to solve the probability flow ordinary differential equation (ODE) defined by an existing diffusion model. CMs are not directly trained to minimize error against an ODE solver, rather they use a more computationally tractable objective. As a way to study how effectively CMs solve the probability flow ODE, and the effect that any induced error has on the quality of generated samples, we introduce Direct CMs, which \textit{directly} minimize this error. Intriguingly, we find that Direct CMs reduce the ODE solving error compared to CMs but also result in significantly worse sample quality, calling into question why exactly CMs work well in the first place. Full code is available at: https://github.com/layer6ai-labs/direct-cms. |
| title | Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2411.08954 |