E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917709277560832 |
|---|---|
| author | Courant, Robin Dufour, Nicolas Wang, Xi Christie, Marc Kalogeiton, Vicky |
| author_facet | Courant, Robin Dufour, Nicolas Wang, Xi Christie, Marc Kalogeiton, Vicky |
| contents | Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this paper, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character information and textual captions encompassing descriptions of both camera and character. To our knowledge, this is the first dataset of its kind. To show the potential applications of the E.T. dataset, we propose a diffusion-based approach, named DIRECTOR, which generates complex camera trajectories from textual captions that describe the relation and synchronisation between the camera and characters. To ensure robust and accurate evaluations, we train on the E.T. dataset CLaTr, a Contrastive Language-Trajectory embedding for evaluation metrics. We posit that our proposed dataset and method significantly advance the democratization of cinematography, making it more accessible to common users. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01516 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness Courant, Robin Dufour, Nicolas Wang, Xi Christie, Marc Kalogeiton, Vicky Computer Vision and Pattern Recognition Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this paper, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character information and textual captions encompassing descriptions of both camera and character. To our knowledge, this is the first dataset of its kind. To show the potential applications of the E.T. dataset, we propose a diffusion-based approach, named DIRECTOR, which generates complex camera trajectories from textual captions that describe the relation and synchronisation between the camera and characters. To ensure robust and accurate evaluations, we train on the E.T. dataset CLaTr, a Contrastive Language-Trajectory embedding for evaluation metrics. We posit that our proposed dataset and method significantly advance the democratization of cinematography, making it more accessible to common users. |
| title | E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.01516 |