Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.22554 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916817755176960 |
|---|---|
| author | Agrawal, Vasu Akinyemi, Akinniyi Alvero, Kathryn Behrooz, Morteza Buffalini, Julia Carlucci, Fabio Maria Chen, Joy Chen, Junming Chen, Zhang Cheng, Shiyang Chowdary, Praveen Chuang, Joe D'Avirro, Antony Daly, Jon Dong, Ning Duppenthaler, Mark Gao, Cynthia Girard, Jeff Gleize, Martin Gomez, Sahir Gong, Hongyu Govindarajan, Srivathsan Han, Brandon He, Sen Hernandez, Denise Hristov, Yordan Huang, Rongjie Inaguma, Hirofumi Jain, Somya Janardhan, Raj Jia, Qingyao Klaiber, Christopher Kovachev, Dejan Kumar, Moneish Li, Hang Li, Yilei Litvin, Pavel Liu, Wei Ma, Guangyao Ma, Jing Ma, Martin Ma, Xutai Mantovani, Lucas Miglani, Sagar Mohan, Sreyas Morency, Louis-Philippe Ng, Evonne Ng, Kam-Woh Nguyen, Tu Anh Oberai, Amia Peloquin, Benjamin Pino, Juan Popovic, Jovan Poursaeed, Omid Prada, Fabian Rakotoarison, Alice Ranjan, Rakesh Richard, Alexander Ropers, Christophe Saleem, Safiyyah Sharma, Vasu Shcherbyna, Alex Shen, Jia Shen, Jie Stathopoulos, Anastasis Sun, Anna Tomasello, Paden Tran, Tuan Turkatenko, Arina Wan, Bo Wang, Chao Wang, Jeff Williamson, Mary Wood, Carleigh Xiang, Tao Yang, Yilin Yao, Julien Zhang, Chen Zhang, Jiemin Zhang, Xinyue Zheng, Jason Zhyzheria, Pavlo Zikes, Jan Zollhoefer, Michael |
| author_facet | Agrawal, Vasu Akinyemi, Akinniyi Alvero, Kathryn Behrooz, Morteza Buffalini, Julia Carlucci, Fabio Maria Chen, Joy Chen, Junming Chen, Zhang Cheng, Shiyang Chowdary, Praveen Chuang, Joe D'Avirro, Antony Daly, Jon Dong, Ning Duppenthaler, Mark Gao, Cynthia Girard, Jeff Gleize, Martin Gomez, Sahir Gong, Hongyu Govindarajan, Srivathsan Han, Brandon He, Sen Hernandez, Denise Hristov, Yordan Huang, Rongjie Inaguma, Hirofumi Jain, Somya Janardhan, Raj Jia, Qingyao Klaiber, Christopher Kovachev, Dejan Kumar, Moneish Li, Hang Li, Yilei Litvin, Pavel Liu, Wei Ma, Guangyao Ma, Jing Ma, Martin Ma, Xutai Mantovani, Lucas Miglani, Sagar Mohan, Sreyas Morency, Louis-Philippe Ng, Evonne Ng, Kam-Woh Nguyen, Tu Anh Oberai, Amia Peloquin, Benjamin Pino, Juan Popovic, Jovan Poursaeed, Omid Prada, Fabian Rakotoarison, Alice Ranjan, Rakesh Richard, Alexander Ropers, Christophe Saleem, Safiyyah Sharma, Vasu Shcherbyna, Alex Shen, Jia Shen, Jie Stathopoulos, Anastasis Sun, Anna Tomasello, Paden Tran, Tuan Turkatenko, Arina Wan, Bo Wang, Chao Wang, Jeff Williamson, Mary Wood, Carleigh Xiang, Tao Yang, Yilin Yao, Julien Zhang, Chen Zhang, Jiemin Zhang, Xinyue Zheng, Jason Zhyzheria, Pavlo Zikes, Jan Zollhoefer, Michael |
| contents | Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent AI technologies, it is crucial to develop models that can both comprehend and generate dyadic behavioral dynamics. To this end, we introduce the Seamless Interaction Dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage from over 4,000 participants in diverse contexts. This dataset enables the development of AI technologies that understand dyadic embodied dynamics, unlocking breakthroughs in virtual agents, telepresence experiences, and multimodal content analysis tools. We also develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech. These models can take as input both the speech and visual behavior of their interlocutors. We present a variant with speech from an LLM model and integrations with 2D and 3D rendering methods, bringing us closer to interactive virtual agents. Additionally, we describe controllable variants of our motion models that can adapt emotional responses and expressivity levels, as well as generating more semantically-relevant gestures. Finally, we discuss methods for assessing the quality of these dyadic motion models, which are demonstrating the potential for more intuitive and responsive human-AI interactions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22554 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset Agrawal, Vasu Akinyemi, Akinniyi Alvero, Kathryn Behrooz, Morteza Buffalini, Julia Carlucci, Fabio Maria Chen, Joy Chen, Junming Chen, Zhang Cheng, Shiyang Chowdary, Praveen Chuang, Joe D'Avirro, Antony Daly, Jon Dong, Ning Duppenthaler, Mark Gao, Cynthia Girard, Jeff Gleize, Martin Gomez, Sahir Gong, Hongyu Govindarajan, Srivathsan Han, Brandon He, Sen Hernandez, Denise Hristov, Yordan Huang, Rongjie Inaguma, Hirofumi Jain, Somya Janardhan, Raj Jia, Qingyao Klaiber, Christopher Kovachev, Dejan Kumar, Moneish Li, Hang Li, Yilei Litvin, Pavel Liu, Wei Ma, Guangyao Ma, Jing Ma, Martin Ma, Xutai Mantovani, Lucas Miglani, Sagar Mohan, Sreyas Morency, Louis-Philippe Ng, Evonne Ng, Kam-Woh Nguyen, Tu Anh Oberai, Amia Peloquin, Benjamin Pino, Juan Popovic, Jovan Poursaeed, Omid Prada, Fabian Rakotoarison, Alice Ranjan, Rakesh Richard, Alexander Ropers, Christophe Saleem, Safiyyah Sharma, Vasu Shcherbyna, Alex Shen, Jia Shen, Jie Stathopoulos, Anastasis Sun, Anna Tomasello, Paden Tran, Tuan Turkatenko, Arina Wan, Bo Wang, Chao Wang, Jeff Williamson, Mary Wood, Carleigh Xiang, Tao Yang, Yilin Yao, Julien Zhang, Chen Zhang, Jiemin Zhang, Xinyue Zheng, Jason Zhyzheria, Pavlo Zikes, Jan Zollhoefer, Michael Computer Vision and Pattern Recognition Artificial Intelligence Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent AI technologies, it is crucial to develop models that can both comprehend and generate dyadic behavioral dynamics. To this end, we introduce the Seamless Interaction Dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage from over 4,000 participants in diverse contexts. This dataset enables the development of AI technologies that understand dyadic embodied dynamics, unlocking breakthroughs in virtual agents, telepresence experiences, and multimodal content analysis tools. We also develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech. These models can take as input both the speech and visual behavior of their interlocutors. We present a variant with speech from an LLM model and integrations with 2D and 3D rendering methods, bringing us closer to interactive virtual agents. Additionally, we describe controllable variants of our motion models that can adapt emotional responses and expressivity levels, as well as generating more semantically-relevant gestures. Finally, we discuss methods for assessing the quality of these dyadic motion models, which are demonstrating the potential for more intuitive and responsive human-AI interactions. |
| title | Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2506.22554 |