Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929355361353728 |
|---|---|
| author | Tu, Hanzhang Shao, Ruizhi Dong, Xue Zheng, Shunyuan Zhang, Hao Chen, Lili Wang, Meili Li, Wenyu Ma, Siyan Zhang, Shengping Zhou, Boyao Liu, Yebin |
| author_facet | Tu, Hanzhang Shao, Ruizhi Dong, Xue Zheng, Shunyuan Zhang, Hao Chen, Lili Wang, Meili Li, Wenyu Ma, Siyan Zhang, Shengping Zhou, Boyao Liu, Yebin |
| contents | In this paper, we present a low-budget and high-authenticity bidirectional telepresence system, Tele-Aloha, targeting peer-to-peer communication scenarios. Compared to previous systems, Tele-Aloha utilizes only four sparse RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achieve high-resolution (2048x2048), real-time (30 fps), low-latency (less than 150ms) and robust distant communication. As the core of Tele-Aloha, we propose an efficient novel view synthesis algorithm for upper-body. Firstly, we design a cascaded disparity estimator for obtaining a robust geometry cue. Additionally a neural rasterizer via Gaussian Splatting is introduced to project latent features onto target view and to decode them into a reduced resolution. Further, given the high-quality captured data, we leverage weighted blending mechanism to refine the decoded image into the final resolution of 2K. Exploiting world-leading autostereoscopic display and low-latency iris tracking, users are able to experience a strong three-dimensional sense even without any wearable head-mounted display device. Altogether, our telepresence system demonstrates the sense of co-presence in real-life experiments, inspiring the next generation of communication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14866 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras Tu, Hanzhang Shao, Ruizhi Dong, Xue Zheng, Shunyuan Zhang, Hao Chen, Lili Wang, Meili Li, Wenyu Ma, Siyan Zhang, Shengping Zhou, Boyao Liu, Yebin Computer Vision and Pattern Recognition In this paper, we present a low-budget and high-authenticity bidirectional telepresence system, Tele-Aloha, targeting peer-to-peer communication scenarios. Compared to previous systems, Tele-Aloha utilizes only four sparse RGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achieve high-resolution (2048x2048), real-time (30 fps), low-latency (less than 150ms) and robust distant communication. As the core of Tele-Aloha, we propose an efficient novel view synthesis algorithm for upper-body. Firstly, we design a cascaded disparity estimator for obtaining a robust geometry cue. Additionally a neural rasterizer via Gaussian Splatting is introduced to project latent features onto target view and to decode them into a reduced resolution. Further, given the high-quality captured data, we leverage weighted blending mechanism to refine the decoded image into the final resolution of 2K. Exploiting world-leading autostereoscopic display and low-latency iris tracking, users are able to experience a strong three-dimensional sense even without any wearable head-mounted display device. Altogether, our telepresence system demonstrates the sense of co-presence in real-life experiments, inspiring the next generation of communication. |
| title | Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.14866 |