LUCY: Linguistic Understanding and Control Yielding Early Stage of Her
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929688248582144 |
|---|---|
| author | Gao, Heting Shao, Hang Wang, Xiong Qiu, Chaofan Shen, Yunhang Cai, Siqi Shi, Yuchen Xu, Zihan Long, Zuwei Zhang, Yike Dong, Shaoqi Fu, Chaoyou Li, Ke Ma, Long Sun, Xing |
| author_facet | Gao, Heting Shao, Hang Wang, Xiong Qiu, Chaofan Shen, Yunhang Cai, Siqi Shi, Yuchen Xu, Zihan Long, Zuwei Zhang, Yike Dong, Shaoqi Fu, Chaoyou Li, Ke Ma, Long Sun, Xing |
| contents | The film Her features Samantha, a sophisticated AI audio agent who is capable of understanding both linguistic and paralinguistic information in human speech and delivering real-time responses that are natural, informative and sensitive to emotional subtleties. Moving one step toward more sophisticated audio agent from recent advancement in end-to-end (E2E) speech systems, we propose LUCY, a E2E speech model that (1) senses and responds to user's emotion, (2) deliver responses in a succinct and natural style, and (3) use external tool to answer real-time inquiries. Experiment results show that LUCY is better at emotion control than peer models, generating emotional responses based on linguistic emotional instructions and responding to paralinguistic emotional cues. Lucy is also able to generate responses in a more natural style, as judged by external language models, without sacrificing much performance on general question answering. Finally, LUCY can leverage function calls to answer questions that are out of its knowledge scope. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16327 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LUCY: Linguistic Understanding and Control Yielding Early Stage of Her Gao, Heting Shao, Hang Wang, Xiong Qiu, Chaofan Shen, Yunhang Cai, Siqi Shi, Yuchen Xu, Zihan Long, Zuwei Zhang, Yike Dong, Shaoqi Fu, Chaoyou Li, Ke Ma, Long Sun, Xing Computation and Language Sound Audio and Speech Processing The film Her features Samantha, a sophisticated AI audio agent who is capable of understanding both linguistic and paralinguistic information in human speech and delivering real-time responses that are natural, informative and sensitive to emotional subtleties. Moving one step toward more sophisticated audio agent from recent advancement in end-to-end (E2E) speech systems, we propose LUCY, a E2E speech model that (1) senses and responds to user's emotion, (2) deliver responses in a succinct and natural style, and (3) use external tool to answer real-time inquiries. Experiment results show that LUCY is better at emotion control than peer models, generating emotional responses based on linguistic emotional instructions and responding to paralinguistic emotional cues. Lucy is also able to generate responses in a more natural style, as judged by external language models, without sacrificing much performance on general question answering. Finally, LUCY can leverage function calls to answer questions that are out of its knowledge scope. |
| title | LUCY: Linguistic Understanding and Control Yielding Early Stage of Her |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.16327 |