Convergence Analysis of Flow Matching in Latent Space with Transformers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910426149683200 |
|---|---|
| author | Jiao, Yuling Lai, Yanming Wang, Yang Yan, Bokai |
| author_facet | Jiao, Yuling Lai, Yanming Wang, Yang Yan, Bokai |
| contents | We present theoretical convergence guarantees for ODE-based generative models, specifically flow matching. We use a pre-trained autoencoder network to map high-dimensional original inputs to a low-dimensional latent space, where a transformer network is trained to predict the velocity field of the transformation from a standard normal distribution to the target latent distribution. Our error analysis demonstrates the effectiveness of this approach, showing that the distribution of samples generated via estimated ODE flow converges to the target distribution in the Wasserstein-2 distance under mild and practical assumptions. Furthermore, we show that arbitrary smooth functions can be effectively approximated by transformer networks with Lipschitz continuity, which may be of independent interest. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_02538 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Convergence Analysis of Flow Matching in Latent Space with Transformers Jiao, Yuling Lai, Yanming Wang, Yang Yan, Bokai Machine Learning We present theoretical convergence guarantees for ODE-based generative models, specifically flow matching. We use a pre-trained autoencoder network to map high-dimensional original inputs to a low-dimensional latent space, where a transformer network is trained to predict the velocity field of the transformation from a standard normal distribution to the target latent distribution. Our error analysis demonstrates the effectiveness of this approach, showing that the distribution of samples generated via estimated ODE flow converges to the target distribution in the Wasserstein-2 distance under mild and practical assumptions. Furthermore, we show that arbitrary smooth functions can be effectively approximated by transformer networks with Lipschitz continuity, which may be of independent interest. |
| title | Convergence Analysis of Flow Matching in Latent Space with Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2404.02538 |