InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909425597415424 |
|---|---|
| author | Zhang, Pan Dong, Xiaoyi Cao, Yuhang Zang, Yuhang Qian, Rui Wei, Xilin Chen, Lin Li, Yifei Niu, Junbo Ding, Shuangrui Guo, Qipeng Duan, Haodong Chen, Xin Lv, Han Nie, Zheng Zhang, Min Wang, Bin Zhang, Wenwei Zhang, Xinyue Ge, Jiaye Li, Wei Li, Jingwen Tu, Zhongying He, Conghui Zhang, Xingcheng Chen, Kai Qiao, Yu Lin, Dahua Wang, Jiaqi |
| author_facet | Zhang, Pan Dong, Xiaoyi Cao, Yuhang Zang, Yuhang Qian, Rui Wei, Xilin Chen, Lin Li, Yifei Niu, Junbo Ding, Shuangrui Guo, Qipeng Duan, Haodong Chen, Xin Lv, Han Nie, Zheng Zhang, Min Wang, Bin Zhang, Wenwei Zhang, Xinyue Ge, Jiaye Li, Wei Li, Jingwen Tu, Zhongying He, Conghui Zhang, Xingcheng Chen, Kai Qiao, Yu Lin, Dahua Wang, Jiaqi |
| contents | Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09596 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Zhang, Pan Dong, Xiaoyi Cao, Yuhang Zang, Yuhang Qian, Rui Wei, Xilin Chen, Lin Li, Yifei Niu, Junbo Ding, Shuangrui Guo, Qipeng Duan, Haodong Chen, Xin Lv, Han Nie, Zheng Zhang, Min Wang, Bin Zhang, Wenwei Zhang, Xinyue Ge, Jiaye Li, Wei Li, Jingwen Tu, Zhongying He, Conghui Zhang, Xingcheng Chen, Kai Qiao, Yu Lin, Dahua Wang, Jiaqi Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time. |
| title | InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.09596 |