AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929468031893504 |
|---|---|
| author | Ni, Minheng Wu, Chenfei Yuan, Huaying Yang, Zhengyuan Gong, Ming Wang, Lijuan Liu, Zicheng Zuo, Wangmeng Duan, Nan |
| author_facet | Ni, Minheng Wu, Chenfei Yuan, Huaying Yang, Zhengyuan Gong, Ming Wang, Lijuan Liu, Zicheng Zuo, Wangmeng Duan, Nan |
| contents | With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to generate multi-sensory outputs has not been fully explored, limiting the application on high-value scenarios such as of directing a film. Developing a movie director agent faces two major challenges: (1) Lack of parallelism and online scheduling with production steps: In the production of multi-sensory films, there are complex dependencies between different sensory elements, and the production time for each element varies. (2) Diverse needs and clear communication demands with users: Users often cannot clearly express their needs until they see a draft, which requires human-computer interaction and iteration to continually adjust and optimize the film content based on user feedback. To address these issues, we introduce AutoDirector, an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. This framework improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs. AutoDirector not only expands the application scope of human-machine collaboration but also demonstrates the potential of AI in collaborating with humans in the role of a film director to complete multi-sensory films. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11564 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition Ni, Minheng Wu, Chenfei Yuan, Huaying Yang, Zhengyuan Gong, Ming Wang, Lijuan Liu, Zicheng Zuo, Wangmeng Duan, Nan Computer Vision and Pattern Recognition With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to generate multi-sensory outputs has not been fully explored, limiting the application on high-value scenarios such as of directing a film. Developing a movie director agent faces two major challenges: (1) Lack of parallelism and online scheduling with production steps: In the production of multi-sensory films, there are complex dependencies between different sensory elements, and the production time for each element varies. (2) Diverse needs and clear communication demands with users: Users often cannot clearly express their needs until they see a draft, which requires human-computer interaction and iteration to continually adjust and optimize the film content based on user feedback. To address these issues, we introduce AutoDirector, an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. This framework improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs. AutoDirector not only expands the application scope of human-machine collaboration but also demonstrates the potential of AI in collaborating with humans in the role of a film director to complete multi-sensory films. |
| title | AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.11564 |